Original: G RAPH QA: P ROTEIN M ODEL Q UALITY A SSESSMENT USING G RAPH C ONVOLUTIONAL N ETWORK
Revised: G RAPH QA: P ROTEIN M ODEL Q UALITY A SSESSMENT USING G RAPH C ONVOLUTIONAL N ETWORK

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Proteins are ubiquitous molecules whose function in biological processes is determined by their 3D structure.
Revised: Proteins are ubiquitous molecules whose function in biological processes is determined by their 3D structure.

Original: Experimental identiﬁcation of a protein’s structure can be time-consuming, prohibitively expensive, and not always possible.
Revised: Experimental identiﬁcation of a protein’s structure can be time-consuming, prohibitively expensive, and not always possible.

Original: Alternatively, protein folding can be modeled using computational methods, which however are not guaranteed to always produce optimal results.
Revised: Alternatively, protein folding can be modeled using computational methods, which however are not guaranteed to always produce optimal results.

Original: G RAPH QA is a graph-based method to estimate the quality of protein models, that possesses favorable properties such as representation learning, explicit modeling of both sequential and 3D structure, geometric invariance and computational efﬁciency.
Revised: G RAPH QA is a graph-based method to estimate the quality of protein models, that possesses favorable properties such as representation learning, explicit modeling of both sequential and 3D structure, geometric invariance and computational efﬁciency.

Original: In this work, we demonstrate signiﬁcant improvements over the state-ofthe-art for both hand-engineered and representation-learning approaches, as well as carefully evaluating the individual contributions of G RAPH QA components.
Revised: In this work, we demonstrate signiﬁcant improvements over the state-ofthe-art for both hand-engineered and representation-learning approaches, as well as carefully evaluating the individual contributions of G RAPH QA components.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Protein molecules are predominantly present in biological forms, responsible for their cellular functions.
Revised: Protein molecules are predominantly present in biological forms, responsible for their cellular functions.

Original: Therefore, understanding, predicting and modifying proteins in biological processes are essential for medical, pharmaceutical and genetic research.
Revised: Therefore, understanding, predicting and modifying proteins in biological processes are essential for medical, pharmaceutical and genetic research.

Original: Such studies strongly depend on discovering mechanical and chemical properties of proteins through the determination of their structure.
Revised: Such studies strongly depend on discovering mechanical and chemical properties of proteins through the determination of their structure.

Original: At the high level, a protein molecule is a chain of hundreds of smaller molecules called amino acids.
Revised: At the high level, a protein molecule is a chain of hundreds of smaller molecules called amino acids.

Original: Identifying a protein’s amino-acid sequence is nowadays straightforward.
Revised: Identifying a protein’s amino-acid sequence is nowadays straightforward.

Original: However, the function of a protein is primarily determined by its 3D structure.
Revised: However, the function of a protein is primarily determined by its 3D structure.

Original: Spatial folding can be determined experimentally, but the existing procedures are time-consuming, prohibitively expensive and not always possible.
Revised: Spatial folding can be determined experimentally, but the existing procedures are time-consuming, prohibitively expensive and not always possible.

Original: Thus, several computational techniques were developed for protein structure prediction (Arnold et al., 2006;  Xu, 2019).
Revised: Thus, several computational techniques were developed for protein structure prediction (Arnold et al., 2006; Wang et al., 2017; Xu, 2019).

Original: So far, no single method is always best, e.g. some protein families are best modeled by certain methods, also, computational methods often produce multiple outputs.
Revised: So far, no single method is always best, e.g. some protein families are best modeled by certain methods, also, computational methods often produce multiple outputs.

Original: Therefore, candidate generation is generally followed by an evaluation step.
Revised: Therefore, candidate generation is generally followed by an evaluation step.

Original: This work focuses on Quality Assessment (QA) of computationally-derived models of a protein (Lundstrom et al., 2001; Won et al., 2019).
Revised: This work focuses on Quality Assessment (QA) of computationally-derived models of a protein (Lundstrom et al., 2001; Won et al., 2019).

Original: QA, also referred to as model accuracy estimation (MAE), estimates the quality of computational protein models in terms of divergence from their native structure.
Revised: QA, also referred to as model accuracy estimation (MAE), estimates the quality of computational protein models in terms of divergence from their native structure.

Original: The downstream goal of QA is two-fold: to ﬁnd the best model in a pool of models and to reﬁne a model based on its local quality.
Revised: The downstream goal of QA is two-fold: to ﬁnd the best model in a pool of models and to reﬁne a model based on its local quality.

Original: Computational protein folding and design has recently received attention from the machine learning community (Xu, 2019; Jones & Kandathil, 2018;Ingraham et al., 2019b; Anand & Huang, 2018;
Revised: Computational protein folding and design has recently received attention from the machine learning community (Wang et al., 2017; Xu, 2019; Jones & Kandathil, 2018;

Original:  Evans et al., 2018; AlQuraishi, 2019), while QA has yet to follow.
Revised: Ingraham et al., 2019b; Anand & Huang, 2018; Evans et al., 2018; AlQuraishi, 2019), while QA has yet to follow.

Original: This is despite the importance of QA for structural biology and the availability of standard datasets to benchmark machine learning techniques, such as the biannual CASP event (Moult et al., 1999).
Revised: This is despite the importance of QA for structural biology and the availability of standard datasets to benchmark machine learning techniques, such as the biannual CASP event (Moult et al., 1999).

Original: The ﬁeld of bioinformatics, on the other hand, has witnessed noticeable progress in QA for more than a decade: from earlier works using artiﬁcial neural networks (Wallner & Elofsson, 2006) or support vector machines (Ray et al., 2012; Uziela et al., 2016) to more recent deep learning methods based on 1D-CNNs, 3D-CNNs and 
Revised: The ﬁeld of bioinformatics, on the other hand, has witnessed noticeable progress in QA for more than a decade: from earlier works using artiﬁcial neural networks (Wallner & Elofsson, 2006) or support vector machines (Ray et al., 2012; Uziela et al., 2016) to more recent deep learning methods based on 1DCNNs, 3D-CNNs and LSTMs (Hurtado et al., 2018; Derevyanko et al., 2018;

Original: LSTMs (Hurtado et al., 2018; Derevyanko et al., 2018; Pag`es et al., 2018; Conover et al., 2019).
Revised: Pag`es et al., 2018; Conover et al., 2019).

Original: In this work, we tackle Quality Assessment with Graph Convolutional Networks, which offer several desirable properties over previous methods.
Revised: In this work, we tackle Quality Assessment with Graph Convolutional Networks, which offer several desirable properties over previous methods.

Original: Through extensive experiments, we show signiﬁcant improvements over the state-of-the-art, and offer informative qualitative and quantitative analyses.
Revised: Through extensive experiments, we show signiﬁcant improvements over the state-of-the-art, and offer informative qualitative and quantitative analyses.

Original: CASP    native               →    →    →  { }    { }   , 
Revised: CASP    native               →    →    →  { }    { }   ,

Original: ENCODER
Revised: ENCODER

Original: MESSAGE PASSING
Revised: MESSAGE PASSING

Original: READOUT ×  { }  ℓ    
Revised: READOUT ×  { }  ℓ

Original: 1. R ELATED W ORKS
Revised: 1. R ELATED W ORKS

Original: Protein Quality Assessment (QA) methods are evaluated in CASP (Moult et al., 1995) since
Revised: Protein Quality Assessment (QA) methods are evaluated in CASP (Moult et al., 1995) since

Original: CASP7 (Cozzetto et al., 2007).
Revised: CASP7 (Cozzetto et al., 2007).

Original: Current techniques can be divided into two categories: singlemodel methods which operate on a single protein model to estimate its quality (Wallner & Elofsson, 2003), and consensus methods that use consistency between several candidates to estimate their quality (Lundstrom et al., 2001).
Revised: Current techniques can be divided into two categories: singlemodel methods which operate on a single protein model to estimate its quality (Wallner & Elofsson, 2003), and consensus methods that use consistency between several candidates to estimate their quality (Lundstrom et al., 2001).

Original: Single-model methods are applicable to a single protein in isolation and in the recent CASP13 performed comparable to or better than consensus methods for the ﬁrst time (Cheng et al., 2019).
Revised: Single-model methods are applicable to a single protein in isolation and in the recent CASP13 performed comparable to or better than consensus methods for the ﬁrst time (Cheng et al., 2019).

Original: Recent single-model QA works are based on deep learning, except VoroMQA that takes a statistical approach on atom-level contact area (Olechnovic & Venclovas, 2017).
Revised: Recent single-model QA works are based on deep learning, except VoroMQA that takes a statistical approach on atom-level contact area (Olechnovic & Venclovas, 2017).

Original: 3DCNN adopts a volumetric representation of proteins (Derevyanko et al., 2018).
Revised: 3DCNN adopts a volumetric representation of proteins (Derevyanko et al., 2018).

Original: Ornate improves 3DCNN by deﬁning a canonical orientation (Pag`es et al., 2018).
Revised: Ornate improves 3DCNN by deﬁning a canonical orientation (Pag`es et al., 2018).

Original: ProQ3D uses a multi-layer perceptron on ﬁxed-length protein descriptors (Uziela et al., 2017).
Revised: ProQ3D uses a multi-layer perceptron on ﬁxed-length protein descriptors (Uziela et al., 2017).

Original: ProQ4 adopts a pre-trained 1D-CNN that is ﬁne-tuned in a siamese conﬁguration with a rank loss (Hurtado et al., 2018).
Revised: ProQ4 adopts a pre-trained 1D-CNN that is ﬁne-tuned in a siamese conﬁguration with a rank loss (Hurtado et al., 2018).

Original: VoroMQA and ProQ3D are among the top performers of CASP13 (Won et al., 2019).
Revised: VoroMQA and ProQ3D are among the top performers of CASP13 (Won et al., 2019).

Original: Graph Convolutional Networks (GCNs) bring the representation learning power of CNNs to graph data, and have been recently applied with success to multiple domains, e.g. physics (Gonzalez et al., 2018), visual scene understanding (Narasimhan et al., 2018) and natural language understanding (Kipf & Welling, 2017).
Revised: Graph Convolutional Networks (GCNs) bring the representation learning power of CNNs to graph data, and have been recently applied with success to multiple domains, e.g. physics (Gonzalez et al., 2018), visual scene understanding (Narasimhan et al., 2018) and natural language understanding (Kipf & Welling, 2017).

Original: Molecules can be naturally represented as graphs and GCNs have been proven effective in several related tasks, including molecular representation learning (Duvenaud et al., 2015), protein interface prediction (Fout et al., 2017), chemical property prediction (Niepert et al., 2016;
Revised: Molecules can be naturally represented as graphs and GCNs have been proven effective in several related tasks, including molecular representation learning (Duvenaud et al., 2015), protein interface prediction (Fout et al., 2017), chemical property prediction (Niepert et al., 2016;

Original: Gilmer et al., 2017; Li et al., 2018a), drug-drug interaction (Zitnik et al., 2018), drugtarget interaction (Gao et al., 2018) molecular optimization (Jin et al., 2019), and generation of proteins, molecules and drugs (Ingraham et al., 2019a;
Revised: Gilmer et al., 2017; Li et al., 2018a), drug-drug interaction (Zitnik et al., 2018), drugtarget interaction (Gao et al., 2018) molecular optimization (Jin et al., 2019), and generation of proteins, molecules and drugs (Ingraham et al., 2019a;

Original: You et al., 2018; Liu et al., 2018;
Revised: You et al., 2018; Liu et al., 2018;

Original: Li et al., 2018b; Simonovsky & Komodakis, 2018).
Revised: Li et al., 2018b; Simonovsky & Komodakis, 2018).

Original: However, to the best of our knowledge, GCNs have never been applied to the problem of protein quality assessment.
Revised: However, to the best of our knowledge, GCNs have never been applied to the problem of protein quality assessment.

Original: 1. C ONTRIBUTIONS
Revised: 1. C ONTRIBUTIONS

Original: • This work is the ﬁrst to tackle QA with GCN which bring several desirable properties over previous methods, including representation learning (3DCNN, Ornate), geometric invariance (VoroMQA, Ornate), sequence learning (ProQ4, AngularQA), explicit modeling of 3D structure (3DCNN, Ornate, VoroMQA) and computational efﬁciency.
Revised: • This work is the ﬁrst to tackle QA with GCN which bring several desirable properties over previous methods, including representation learning (3DCNN, Ornate), geometric invariance (VoroMQA, Ornate), sequence learning (ProQ4, AngularQA), explicit modeling of 3D structure (3DCNN, Ornate, VoroMQA) and computational efﬁciency.

Original: • Thanks to these desirable properties, a simple GCN setup achieves improved results compared to the more sophisticated state-of-the-art methods such as ProQ4.
Revised: • Thanks to these desirable properties, a simple GCN setup achieves improved results compared to the more sophisticated state-of-the-art methods such as ProQ4.

Original: This is demonstrated through extensive experiments on multiple datasets and scoring regimes.
Revised: This is demonstrated through extensive experiments on multiple datasets and scoring regimes.

Original: • Novel representation techniques are employed to explicitly reﬂect the sequential (residue separation) and 3D structure (angles, spatial distance and secondary structure) of proteins.
Revised: • Novel representation techniques are employed to explicitly reﬂect the sequential (residue separation) and 3D structure (angles, spatial distance and secondary structure) of proteins.

Original: • Enabled by the use of GCN, we combine the optimization of local and global score for QA, improving over the performance of a global-only scoring method.
Revised: • Enabled by the use of GCN, we combine the optimization of local and global score for QA, improving over the performance of a global-only scoring method.

Original: • Through an extensive set of ablation studies, the signiﬁcance
Revised: • Through an extensive set of ablation studies, the signiﬁcance

Original: of different components of the method, including architecture, loss, and features, are carefully analyzed.
Revised: of different components of the method, including architecture, loss, and features, are carefully analyzed.

Original: PyTorch implementation, datasets and experiments in this anonymized repo: https://bit.ly/36UqISj
Revised: PyTorch implementation, datasets and experiments in this anonymized repo: https://bit.ly/36UqISj

Original:  M ETHOD
Revised:  M ETHOD

Original: We start describing our method by arguing for representation of protein molecules as graphs in learning tasks, then we deﬁne the problem of protein quality assessment (QA), and ﬁnally we present the proposed G RAPH QA architecture.
Revised: We start describing our method by arguing for representation of protein molecules as graphs in learning tasks, then we deﬁne the problem of protein quality assessment (QA), and ﬁnally we present the proposed G RAPH QA architecture.

Original: 2. P ROTEIN REPRESENTATION AS G RAPHS
Revised: 2. P ROTEIN REPRESENTATION AS G RAPHS

Original: Proteins are large molecular structures that perform vital functions in all living organisms.
Revised: Proteins are large molecular structures that perform vital functions in all living organisms.

Original: At the chemical level, a protein consists of one or more chains of smaller molecules, which we interchangeably refer to as residues for their role in the chain, or as amino acids for their chemical composition.
Revised: At the chemical level, a protein consists of one or more chains of smaller molecules, which we interchangeably refer to as residues for their role in the chain, or as amino acids for their chemical composition.

Original: The sequence of residues S = { a i } that composes a protein represents its primary structure , where a i is one of the 22 amino acid types.
Revised: The sequence of residues S = { a i } that composes a protein represents its primary structure , where a i is one of the 22 amino acid types.

Original: The interactions between neighboring residues and with the environment dictate how the chain will fold into complex spatial structures that represent the protein’s secondary structure and tertiary structure .
Revised: The interactions between neighboring residues and with the environment dictate how the chain will fold into complex spatial structures that represent the protein’s secondary structure and tertiary structure .

Original: Therefore, for learning tasks involving proteins, a suitable representation should reﬂect both the identity and sequence of the residues, i.e. the primary structure, and geometric information about the protein’s arrangement in space, i.e. its tertiary structure.
Revised: Therefore, for learning tasks involving proteins, a suitable representation should reﬂect both the identity and sequence of the residues, i.e. the primary structure, and geometric information about the protein’s arrangement in space, i.e. its tertiary structure.

Original: Some works use RNN or 1D-CNN to model proteins as sequence with the spatial structure potentially embedded in the handcrafted residue features (Hurtado et al., 2018; Conover et al., 2019).
Revised: Some works use RNN or 1D-CNN to model proteins as sequence with the spatial structure potentially embedded in the handcrafted residue features (Hurtado et al., 2018; Conover et al., 2019).

Original: Other recent works explicitly model proteins’ spatial structure using 3D-CNN
Revised: Other recent works explicitly model proteins’ spatial structure using 3D-CNN

Original: but ignore its sequential nature (Derevyanko et al., 2018;
Revised: but ignore its sequential nature (Derevyanko et al., 2018;

Original: Pag ` es et al., 2018).
Revised: Pag ` es et al., 2018).

Original: We argue that a graph-based learning can explicitly model both the sequential and geometric structures of proteins.
Revised: We argue that a graph-based learning can explicitly model both the sequential and geometric structures of proteins.

Original: Moreover, it accommodates proteins of different lengths and spatial extent, and is invariant to rotations and translations.
Revised: Moreover, it accommodates proteins of different lengths and spatial extent, and is invariant to rotations and translations.

Original: In the simplest form, a protein can be represented as a linear graph, where nodes represent amino acids and edges connect consecutive residues according to the primary structure.
Revised: In the simplest form, a protein can be represented as a linear graph, where nodes represent amino acids and edges connect consecutive residues according to the primary structure.

Original: This set of edges, which represent the covalent bonds that form the protein backbone, can be extended to include the interactions between non-consecutive residues, e.g. through Van der
Revised: This set of edges, which represent the covalent bonds that form the protein backbone, can be extended to include the interactions between non-consecutive residues, e.g. through Van der

Original: Waals forces or hydrogen bonds, commonly denoted as contacts .
Revised: Waals forces or hydrogen bonds, commonly denoted as contacts .

Original: By forming an edge between all pairs of residues that are within a chemically reasonable distance of each other, the graph becomes a rich representation of both the sequential and geometric structure of the protein (ﬁgure 2).
Revised: By forming an edge between all pairs of residues that are within a chemically reasonable distance of each other, the graph becomes a rich representation of both the sequential and geometric structure of the protein (ﬁgure 2).

Original: We refer to this representation, composed of residues, bonds and contacts, as the protein graph :
Revised: We refer to this representation, composed of residues, bonds and contacts, as the protein graph :

Original: [Equation]
Revised: [Equation]

Original: where i, j = 1 , . .
Revised: where i, j = 1 , . .

Original: . , | S | are residue indices, C = { ( x, y, z ) i } represents the spatial arrangement of the residues, i.e. the protein’s conformation , and d max is a cutoff distance for contacts.
Revised: . , | S | are residue indices, C = { ( x, y, z ) i } represents the spatial arrangement of the residues, i.e. the protein’s conformation , and d max is a cutoff distance for contacts.

Original: With the protein’s structure encoded in the graph, additional residue and relationship features can be encoded as nodes and edges attributes, v i and e i,j respectively.
Revised: With the protein’s structure encoded in the graph, additional residue and relationship features can be encoded as nodes and edges attributes, v i and e i,j respectively.

Original: Section 3.2 describes, in detail, an attribution that preserves the sequence information and 3D geometry while remaining invariant to rotation.
Revised: Section 3.2 describes, in detail, an attribution that preserves the sequence information and 3D geometry while remaining invariant to rotation.

Original: [Figure]
Revised: [Figure]

Original: 2. P ROTEIN Q UALITY A SSESSMENT
Revised: 2. P ROTEIN Q UALITY A SSESSMENT

Original: Experimental identiﬁcation of a protein’s native structure can be time consuming and prohibitively expensive.
Revised: Experimental identiﬁcation of a protein’s native structure can be time consuming and prohibitively expensive.

Original: Alternatively, computational folding methods are used to generate decoy conformations for a speciﬁc target protein.
Revised: Alternatively, computational folding methods are used to generate decoy conformations for a speciﬁc target protein.

Original: Since no single method is consistently best, a Quality Assessment (QA) step is used to identify the decoys that most correctly represent the native structure.
Revised: Since no single method is consistently best, a Quality Assessment (QA) step is used to identify the decoys that most correctly represent the native structure.

Original: If the native structure C native is experimentally determined, the quality of a decoy can be measured by comparison, e.g., in the CASP challenge, decoys submitted for a target are scored against the unreleased native structure.
Revised: If the native structure C native is experimentally determined, the quality of a decoy can be measured by comparison, e.g., in the CASP challenge, decoys submitted for a target are scored against the unreleased native structure.

Original: Some comparative algorithms compute global (per decoy) scores, which can be used for ranking and represent the principal factor for CASP, while others produce local (per residue) scores which help identify incorrect parts of a decoy (Uziela et al., 2018).
Revised: Some comparative algorithms compute global (per decoy) scores, which can be used for ranking and represent the principal factor for CASP, while others produce local (per residue) scores which help identify incorrect parts of a decoy (Uziela et al., 2018).

Original: In most scenarios, however, the native structure is not available and quality must be estimated based on physical and chemical properties of the decoy, e.g. in drug development, it would be unpractical to synthesize samples of novel proteins and researchers rely on computational folding and quality assessment instead.
Revised: In most scenarios, however, the native structure is not available and quality must be estimated based on physical and chemical properties of the decoy, e.g. in drug development, it would be unpractical to synthesize samples of novel proteins and researchers rely on computational folding and quality assessment instead.

Original: Here we introduce G RAPH QA, a graph-based QA neural network that learns to predict local and global scores, with minimal feature and model engineering, using existing datasets of scored proteins.
Revised: Here we introduce G RAPH QA, a graph-based QA neural network that learns to predict local and global scores, with minimal feature and model engineering, using existing datasets of scored proteins.

Original: In this paper, we train G RAPH QA on two widely-used scoring algorithms: the Global Distance
Revised: In this paper, we train G RAPH QA on two widely-used scoring algorithms: the Global Distance

Original: Test Total Score (Zemla, 2003), which is the ofﬁcial CASP score for decoy-level quality assessment, and the Local Distance Difference Test (Mariani et al., 2013), a residue-level score.
Revised: Test Total Score (Zemla, 2003), which is the ofﬁcial CASP score for decoy-level quality assessment, and the Local Distance Difference Test (Mariani et al., 2013), a residue-level score.

Original: We denote them as q g :=
Revised: We denote them as q g :=

Original: GDT TS ( C d , C native ) and { q (cid:96)i } := LDDT ( C d , C native ) .
Revised: GDT TS ( C d , C native ) and { q (cid:96)i } := LDDT ( C d , C native ) .

Original: With G RAPH QA (cid:96)i ( P ) and G RAPH QA g ( P ) denoting the network’s local and global predictions for an input P , the learning objective is to minimize the following Mean Squared Error (MSE) losses:
Revised: With G RAPH QA (cid:96)i ( P ) and G RAPH QA g ( P ) denoting the network’s local and global predictions for an input P , the learning objective is to minimize the following Mean Squared Error (MSE) losses:

Original: [Equation]
Revised: [Equation]

Original: Note that, for the sole purpose of sorting decoy according to ground-truth quality, training with a ranking loss would be sufﬁcient (Derevyanko et al., 2018).
Revised: Note that, for the sole purpose of sorting decoy according to ground-truth quality, training with a ranking loss would be sufﬁcient (Derevyanko et al., 2018).

Original: Instead, MSE forces the output to match the quality score, which is a harder objective, but results in a network can be more easily inspected and possibly used to improve existing folding methods in an end-to-end fashion (section 4.2).
Revised: Instead, MSE forces the output to match the quality score, which is a harder objective, but results in a network can be more easily inspected and possibly used to improve existing folding methods in an end-to-end fashion (section 4.2).

Original: 2. G RAPH QA A RCHITECTURE
Revised: 2. G RAPH QA A RCHITECTURE

Original: G RAPH QA is a graph convolutional network that operates on protein graphs using the messagepassing algorithm described in Battaglia et al.
Revised: G RAPH QA is a graph convolutional network that operates on protein graphs using the messagepassing algorithm described in Battaglia et al.

Original: The building block of G RAPH QA, a graph layer, takes a protein graph as input (with an additional global feature u ), and performs the following propagation steps to output a graph with updated node/edge/global features and unchanged structure:
Revised: The building block of G RAPH QA, a graph layer, takes a protein graph as input (with an additional global feature u ), and performs the following propagation steps to output a graph with updated node/edge/global features and unchanged structure:

Original: [Equation]
Revised: [Equation]

Original: Update edges
Revised: Update edges

Original: Aggregate edges
Revised: Aggregate edges

Original: Update nodes
Revised: Update nodes

Original: [Equation]
Revised: [Equation]

Original: Aggregate all edges
Revised: Aggregate all edges

Original: Aggregate all nodes
Revised: Aggregate all nodes

Original: Update global features where φ represent three update functions that transform nodes/edges/global features (e.g. a MLP), and ρ represent three pooling functions that aggregate features at various levels (e.g. sum or mean).
Revised: Update global features where φ represent three update functions that transform nodes/edges/global features (e.g. a MLP), and ρ represent three pooling functions that aggregate features at various levels (e.g. sum or mean).

Original: Similarly to CNNs, multiple graph layers are stacked together to allow local information to propagate to increasingly larger neighborhoods (i.e. receptive ﬁeld).
Revised: Similarly to CNNs, multiple graph layers are stacked together to allow local information to propagate to increasingly larger neighborhoods (i.e. receptive ﬁeld).

Original: This enables the network to learn quality-related features at multiple scales: secondary structures in the ﬁrst layers, e.g. α -helices and β -sheets, and larger structures in deeper layers e.g. domain structures and arrangements.
Revised: This enables the network to learn quality-related features at multiple scales: secondary structures in the ﬁrst layers, e.g. α -helices and β -sheets, and larger structures in deeper layers e.g. domain structures and arrangements.

Original: The G RAPH QA architecture is conceptually divided in three stages (ﬁgure 1).
Revised: The G RAPH QA architecture is conceptually divided in three stages (ﬁgure 1).

Original: At the input, the encoder increases the node and edge features’ dimensions through 2 × (Linear-Dropout-ReLU) transformation and adds a global bias.
Revised: At the input, the encoder increases the node and edge features’ dimensions through 2 × (Linear-Dropout-ReLU) transformation and adds a global bias.

Original: Then, at its the core, L message-passing layers operate on the encoded graph, leveraging its structure to propagate and aggregate information.
Revised: Then, at its the core, L message-passing layers operate on the encoded graph, leveraging its structure to propagate and aggregate information.

Original: The update functions φ consist of Linear-Dropout-ReLU transformations, with the size of the linear layers progressively decreasing.
Revised: The update functions φ consist of Linear-Dropout-ReLU transformations, with the size of the linear layers progressively decreasing.

Original: We use average pooling for the aggregation functions ρ , since preliminary experiments with max/sum pooling performed poorly.
Revised: We use average pooling for the aggregation functions ρ , since preliminary experiments with max/sum pooling performed poorly.

Original: Finally, the readout layer outputs local and global quality scores by applying a Linear-Sigmoid operation to the latest node and global features, respectively.
Revised: Finally, the readout layer outputs local and global quality scores by applying a Linear-Sigmoid operation to the latest node and global features, respectively.

Original:  E XPERIMENTS 3. E XPERIMENTAL S ETUP
Revised:  E XPERIMENTS 3. E XPERIMENTAL S ETUP

Original: Following the common practice in Quality Assessment, we use the data from past years’ editions of
Revised: Following the common practice in Quality Assessment, we use the data from past years’ editions of

Original: CASP, encompassing several targets with multiple scored decoys each.
Revised: CASP, encompassing several targets with multiple scored decoys each.

Original: Removing all sequences with | S | < 50 from CASP 7-10 results in a dataset of ∼ 100k scored decoys ( P , { q (cid:96)i } , q g ) t,d , which we randomly split into a training set (402 targets) and a validation set for hyper-parameter optimization (35 targets).
Revised: Removing all sequences with | S | < 50 from CASP 7-10 results in a dataset of ∼ 100k scored decoys ( P , { q (cid:96)i } , q g ) t,d , which we randomly split into a training set (402 targets) and a validation set for hyper-parameter optimization (35 targets).

Original: CASP 11 and 12 are set aside for testing against top-scoring methods (table 3).
Revised: CASP 11 and 12 are set aside for testing against top-scoring methods (table 3).

Original: We evaluate the performances of G RAPH QA on the following standard metrics.
Revised: We evaluate the performances of G RAPH QA on the following standard metrics.

Original: At the global level, we compare the predicted and ground-truth GDT TS scores and compute: Root Mean Squared Error (RMSE), Pearson correlation coefﬁcient computed across all decoys of all targets ( R ), and Pearson correlation coefﬁcient computed on a per-target basis and then averaged over all targets ( R target ).
Revised: At the global level, we compare the predicted and ground-truth GDT TS scores and compute: Root Mean Squared Error (RMSE), Pearson correlation coefﬁcient computed across all decoys of all targets ( R ), and Pearson correlation coefﬁcient computed on a per-target basis and then averaged over all targets ( R target ).

Original: At the local level, we compare the predicted and ground-truth LDDT scores and compute: RMSE,
Revised: At the local level, we compare the predicted and ground-truth LDDT scores and compute: RMSE,

Original: Pearson correlation coefﬁcient computed across all residues of all decoys of all targets ( R ), and
Revised: Pearson correlation coefﬁcient computed across all residues of all decoys of all targets ( R ), and

Original: Pearson correlation coefﬁcient computed on a per-decoy basis and then averaged over all decoys of all targets ( R model ).
Revised: Pearson correlation coefﬁcient computed on a per-decoy basis and then averaged over all decoys of all targets ( R model ).

Original: Of these, we focus on R target and R model , which respectively measure the ability to rank decoys by quality and to distinguish the correctly-predicted parts of a model from those that need improvement.
Revised: Of these, we focus on R target and R model , which respectively measure the ability to rank decoys by quality and to distinguish the correctly-predicted parts of a model from those that need improvement.

Original: A detailed description of these and other metrics can be found in appendix E.
Revised: A detailed description of these and other metrics can be found in appendix E.

Original: 3. F EATURES
Revised: 3. F EATURES

Original: Node features The node attributes v i of a protein graph P represent the identity, statistical, and structural features of the i -th residue.
Revised: Node features The node attributes v i of a protein graph P represent the identity, statistical, and structural features of the i -th residue.

Original: We encode the residue identity using a one-of-22 encoding of the corresponding amino acid.
Revised: We encode the residue identity using a one-of-22 encoding of the corresponding amino acid.

Original: Following Hurtado et al.
Revised: Following Hurtado et al.

Original: (2018), we also add two residue-level statistics computed using Multiple Sequence Alignment (MSA)
Revised: (2018), we also add two residue-level statistics computed using Multiple Sequence Alignment (MSA)

Original: (Rost et al., 1994), namely selfinformation and partial entropy , each described by a 23-dimensional vector.
Revised: (Rost et al., 1994), namely selfinformation and partial entropy , each described by a 23-dimensional vector.

Original: Finally, we add a 14-dimensional vector of 3D spatial information including the dihedral angles, surface accessibility and secondary structure type as determined by DSSP (Kabsch & Sander, 1983).
Revised: Finally, we add a 14-dimensional vector of 3D spatial information including the dihedral angles, surface accessibility and secondary structure type as determined by DSSP (Kabsch & Sander, 1983).

Original: Edge features An edge represents either a contact or a bond between two residues i and j w.r.t.
Revised: Edge features An edge represents either a contact or a bond between two residues i and j w.r.t.

Original: to the conformation C
Revised: to the conformation C

Original: = { ( x, y, z ) i } .
Revised: = { ( x, y, z ) i } .

Original: An edge always exists between two consecutive residues, while non-consecutive residues are only connected
Revised: An edge always exists between two consecutive residues, while non-consecutive residues are only connected

Original: if || C
Revised: if || C

Original: i − C
Revised: i − C

Original: j || <
Revised: j || <

Original: d max , with d max optimized on the validation set.
Revised: d max , with d max optimized on the validation set.

Original: We further enrich this connectivity structure by encoding spatial and sequential distances as an 8D feature vector e i,j .
Revised: We further enrich this connectivity structure by encoding spatial and sequential distances as an 8D feature vector e i,j .

Original: Spatial distance is encoded using a radial basis function exp( − d 2 i,j /σ ) , with σ determined on the validation set.
Revised: Spatial distance is encoded using a radial basis function exp( − d 2 i,j /σ ) , with σ determined on the validation set.

Original: Sequential distance is deﬁned as the number of amino acids between the two residues in the sequence and expressed using a separation encoding , i.e. a one-hot encoding of the separation | i − j | according to the classes { 0 , 1 , 2 , 3 , 4 , 5 : 10 , > 10 } .
Revised: Sequential distance is deﬁned as the number of amino acids between the two residues in the sequence and expressed using a separation encoding , i.e. a one-hot encoding of the separation | i − j | according to the classes { 0 , 1 , 2 , 3 , 4 , 5 : 10 , > 10 } .

Original: 3. O PTIMIZATION AND H YPERPARAMETER S EARCH
Revised: 3. O PTIMIZATION AND H YPERPARAMETER S EARCH

Original: The MSE losses in equation 2 are weighted as L tot = λ (cid:96) L (cid:96) + λ g L g and minimized using Adam
Revised: The MSE losses in equation 2 are weighted as L tot = λ (cid:96) L (cid:96) + λ g L g and minimized using Adam

Original: Optimizer (Kingma & Ba, 2014) with L 2 regularization.
Revised: Optimizer (Kingma & Ba, 2014) with L 2 regularization.

Original: G RAPH QA is signiﬁcantly faster to train than LSTM or 3D-CNN methods, e.g. 35 epochs takes ∼ 2 hours on one NVIDIA 2080Ti
Revised: G RAPH QA is signiﬁcantly faster to train than LSTM or 3D-CNN methods, e.g. 35 epochs takes ∼ 2 hours on one NVIDIA 2080Ti

Original: GPU with batches of 200 graphs.
Revised: GPU with batches of 200 graphs.

Original: This allows us to perform extensive hyper-parameter search.
Revised: This allows us to perform extensive hyper-parameter search.

Original: Table 4 reports the search space, as well as the parameters of the model with highest R target on the validation set.
Revised: Table 4 reports the search space, as well as the parameters of the model with highest R target on the validation set.

Original:  E VALUATION
Revised:  E VALUATION

Original: We compare G RAPH QA with the following methods, chosen either for their state-of-the-art performances or because they represent a class of approaches for Quality Assessment.
Revised: We compare G RAPH QA with the following methods, chosen either for their state-of-the-art performances or because they represent a class of approaches for Quality Assessment.

Original: ProQ3D (Uziela et al., 2017) computes ﬁxed-size statistical descriptions of the decoys in CASP 9-10, including
Revised: ProQ3D (Uziela et al., 2017) computes ﬁxed-size statistical descriptions of the decoys in CASP 9-10, including

Original: Rosetta energy terms, which are then used to train a Multi Layer Perceptron on quality scores.
Revised: Rosetta energy terms, which are then used to train a Multi Layer Perceptron on quality scores.

Original: In
Revised: In

Original: ProQ4 (Hurtado et al., 2018), a 1D-CNN is trained to predict LDDT scores from a vectorized representation of protein sequences, a global score is then obtained by averaging over all residues.
Revised: ProQ4 (Hurtado et al., 2018), a 1D-CNN is trained to predict LDDT scores from a vectorized representation of protein sequences, a global score is then obtained by averaging over all residues.

Original: ProQ is pretrained on a large dataset of protein secondary structures and then ﬁne tuned on CASP 9using a siamese conﬁguration to improve ranking performances.
Revised: ProQ is pretrained on a large dataset of protein secondary structures and then ﬁne tuned on CASP 9using a siamese conﬁguration to improve ranking performances.

Original: Their results are reported on both
Revised: Their results are reported on both

Original: CASP 11, which is used as a validation set, and CASP 12.
Revised: CASP 11, which is used as a validation set, and CASP 12.

Original: 3DCNN (Derevyanko et al., 2018) trains
Revised: 3DCNN (Derevyanko et al., 2018) trains

Original: [Table]
Revised: [Table]

Original: a CNN on a three-dimensional representation of atomic densities to rank the decoys in CASP 7according to their GDT TS scores.
Revised: a CNN on a three-dimensional representation of atomic densities to rank the decoys in CASP 7according to their GDT TS scores.

Original: Notably
Revised: Notably

Original: [Table]
Revised: [Table]

Original: no additional feature is used other than atomic structure and type, however, the ﬁxed-size volumetric representation of this method is sensitive to rotations and does not scale well with protein size.
Revised: no additional feature is used other than atomic structure and type, however, the ﬁxed-size volumetric representation of this method is sensitive to rotations and does not scale well with protein size.

Original: Ornate (Pag ` es et al., 2018) applies a similar 3D approach to predict local CAD-scores (Olechnovic & Venclovas, 2017) and achieves rotation invariance by specifying a canonical residue-centered orientation.
Revised: Ornate (Pag ` es et al., 2018) applies a similar 3D approach to predict local CAD-scores (Olechnovic & Venclovas, 2017) and achieves rotation invariance by specifying a canonical residue-centered orientation.

Original: Although optimized for local scoring, the average of the predicted scores is shown to correlate well with GDT TS.
Revised: Although optimized for local scoring, the average of the predicted scores is shown to correlate well with GDT TS.

Original: AngularQA (Conover et al., 2019) feeds a sequence-like representation of the protein structure to an LSTM to predict GDT TS scores.
Revised: AngularQA (Conover et al., 2019) feeds a sequence-like representation of the protein structure to an LSTM to predict GDT TS scores.

Original: The LSTM network is trained on decoys from 3DRobot and CASP 9-11, while CASPis used for model selection and testing.
Revised: The LSTM network is trained on decoys from 3DRobot and CASP 9-11, while CASPis used for model selection and testing.

Original: VoroMQA and RWplus (Olechnovi ˇ c & Venclovas, 2017;
Revised: VoroMQA and RWplus (Olechnovi ˇ c & Venclovas, 2017;

Original: Zhang & Zhang, 2010) are two statistical potential methods that represent an alternative to the other machine-learning based methods.
Revised: Zhang & Zhang, 2010) are two statistical potential methods that represent an alternative to the other machine-learning based methods.

Original: Table 1 compares the performances of G RAPH QA and other state-of-the-art methods on CASPand 12, while ﬁgure 3 contains a graphical representation of true vs. predicted scores for all target in
Revised: Table 1 compares the performances of G RAPH QA and other state-of-the-art methods on CASPand 12, while ﬁgure 3 contains a graphical representation of true vs. predicted scores for all target in

Original: CASP 12, and an example funnel plot for the decoys of a single target.
Revised: CASP 12, and an example funnel plot for the decoys of a single target.

Original: A more in-depth evaluation on the stage 1 and stage 2 splits of CASP 11, 12, 13 and CAMEO can be found in appendix F.
Revised: A more in-depth evaluation on the stage 1 and stage 2 splits of CASP 11, 12, 13 and CAMEO can be found in appendix F.

Original: Of all methods, only G RAPH QA and ProQ4 co-optimize for local and global predictions, the former thanks to the graph-based architecture, the latter thanks to its siamese training conﬁguration (the results reported for ProQ3D refer to two separate models trained for either local or global scores).
Revised: Of all methods, only G RAPH QA and ProQ4 co-optimize for local and global predictions, the former thanks to the graph-based architecture, the latter thanks to its siamese training conﬁguration (the results reported for ProQ3D refer to two separate models trained for either local or global scores).

Original: At the local level, our method proves to be on par or better than ProQ3D and ProQ4, demonstrating the ability to evaluate quality at the residue level and distinguishing correctly predicted parts of the protein chain.
Revised: At the local level, our method proves to be on par or better than ProQ3D and ProQ4, demonstrating the ability to evaluate quality at the residue level and distinguishing correctly predicted parts of the protein chain.

Original: At the global level, signiﬁcantly higher R and R target metrics indicate than G RAPH QA is more capable than other state-of-the-art methods at ranking decoys based on their overall quality.
Revised: At the global level, signiﬁcantly higher R and R target metrics indicate than G RAPH QA is more capable than other state-of-the-art methods at ranking decoys based on their overall quality.

Original: As shown in our ablation studies (section 5), hand-engineered features like MSA and DSSP contribute to the performances of G RAPH QA, yet we wish to prove that our method can learn directly from raw data.
Revised: As shown in our ablation studies (section 5), hand-engineered features like MSA and DSSP contribute to the performances of G RAPH QA, yet we wish to prove that our method can learn directly from raw data.

Original: G RAPH QA RAW is a variant that relies uniquely on the one-hot encoding of amino acid identity, similarly to how 3D-CNN and Ornate employ atomic features only.
Revised: G RAPH QA RAW is a variant that relies uniquely on the one-hot encoding of amino acid identity, similarly to how 3D-CNN and Ornate employ atomic features only.

Original: The results for G RAPH QA RAW show that, even without additional features, our method outperforms purely representation-learning methods.
Revised: The results for G RAPH QA RAW show that, even without additional features, our method outperforms purely representation-learning methods.

Original: 4. A BLATION S TUDIES
Revised: 4. A BLATION S TUDIES

Original: In this section we analyse how various components of our method contribute to the ﬁnal performance, ranging from optimization and architectural choices to protein feature selection.
Revised: In this section we analyse how various components of our method contribute to the ﬁnal performance, ranging from optimization and architectural choices to protein feature selection.

Original: Unless stated otherwise, all ablation studies follow the training procedure described in section 3.3 for a lower number of epochs.
Revised: Unless stated otherwise, all ablation studies follow the training procedure described in section 3.3 for a lower number of epochs.

Original: We report results on CASP 11 as mean and standard deviation of 10 runs.
Revised: We report results on CASP 11 as mean and standard deviation of 10 runs.

Original: Local and global co-optimization We investigate the interplay between local and global predictions, speciﬁcally whether co-optimizing for both is beneﬁcial or detrimental.
Revised: Local and global co-optimization We investigate the interplay between local and global predictions, speciﬁcally whether co-optimizing for both is beneﬁcial or detrimental.

Original: At the global level, models trained to predict only global scores achieve a global RMSE of 0 .
Revised: At the global level, models trained to predict only global scores achieve a global RMSE of 0 .

Original: 129 ± .
Revised: 129 ± .

Original: 007 , whereas models trained to predict both local and global scores obtain 0 .
Revised: 007 , whereas models trained to predict both local and global scores obtain 0 .

Original: 117 ± .
Revised: 117 ± .

Original: 006 , suggesting that local scores can provide additional information and help the assessment of global quality.
Revised: 006 , suggesting that local scores can provide additional information and help the assessment of global quality.

Original: At the local level instead, co-optimization does not seem to improve performances: models trained uniquely on local scores achieve a local RMSE of 0 .
Revised: At the local level instead, co-optimization does not seem to improve performances: models trained uniquely on local scores achieve a local RMSE of 0 .

Original: 121 ± .
Revised: 121 ± .

Original: 002 , while models trained to predict both obtain 0 .
Revised: 002 , while models trained to predict both obtain 0 .

Original: 123 ± .
Revised: 123 ± .

Original: Connectivity and Architecture
Revised: Connectivity and Architecture

Original: In this study, we test the combined effects of the depth of the network L and
Revised: In this study, we test the combined effects of the depth of the network L and

Original: the cutoff value d max .
Revised: the cutoff value d max .

Original: On the one hand, every additional message-passing layer allows to aggregate information from a neighborhood that is one hop larger than the previous, effectively extending the receptive ﬁeld at the readout.
Revised: On the one hand, every additional message-passing layer allows to aggregate information from a neighborhood that is one hop larger than the previous, effectively extending the receptive ﬁeld at the readout.

Original: On the other hand, the number of contacts included in the graph affects its connectivity and the propagation of messages, e.g. low d max correspond to a low average degree and long shortest paths between any two residues, and vice versa (section B.2).
Revised: On the other hand, the number of contacts included in the graph affects its connectivity and the propagation of messages, e.g. low d max correspond to a low average degree and long shortest paths between any two residues, and vice versa (section B.2).

Original: Thus, an architecture that operates on sparsely-connected graphs will require more message-passing layers to achieve the same holistic view of a shallow network operating on denser representations.
Revised: Thus, an architecture that operates on sparsely-connected graphs will require more message-passing layers to achieve the same holistic view of a shallow network operating on denser representations.

Original: However, this trade off is only properly exposed if u , φ u , ρ u are removed from the architecture.
Revised: However, this trade off is only properly exposed if u , φ u , ρ u are removed from the architecture.

Original: In fact, this global pathway represents a propagation shortcut that connects all nodes in the graph and sidesteps the limitations of shallow networks.
Revised: In fact, this global pathway represents a propagation shortcut that connects all nodes in the graph and sidesteps the limitations of shallow networks.

Original: With the global pathway disabled, global predictions are computed in the readout layer by aggregating node features from the last MP layer.
Revised: With the global pathway disabled, global predictions are computed in the readout layer by aggregating node features from the last MP layer.

Original: Figure 4 reports the RMSE obtained by networks of different depth with no global path, operating on protein graphs constructed with different cutoff values.
Revised: Figure 4 reports the RMSE obtained by networks of different depth with no global path, operating on protein graphs constructed with different cutoff values.

Original: As expected, the shallow 3-layer architecture requires more densely-connected inputs to achieve the same performances of the 9-layer network.
Revised: As expected, the shallow 3-layer architecture requires more densely-connected inputs to achieve the same performances of the 9-layer network.

Original: Surprisingly, local predictions seem to be more affected by these factors than global predictions, suggesting that a large receptive ﬁeld is important even for local scores.
Revised: Surprisingly, local predictions seem to be more affected by these factors than global predictions, suggesting that a large receptive ﬁeld is important even for local scores.

Original: Node and Edge Features We evaluate the impact of node and edge features on the overall prediction performances (ﬁgure 5).
Revised: Node and Edge Features We evaluate the impact of node and edge features on the overall prediction performances (ﬁgure 5).

Original: For the nodes, we use the amino acid identity as a minimal representation and combine it with: a) DSSP features, b) partial entropy, c) self information, d) both
Revised: For the nodes, we use the amino acid identity as a minimal representation and combine it with: a) DSSP features, b) partial entropy, c) self information, d) both

Original: DSSP and MSA features.
Revised: DSSP and MSA features.

Original: All features improve both local and global scoring, with DSSP features being marginally more relevant for LDDT.
Revised: All features improve both local and global scoring, with DSSP features being marginally more relevant for LDDT.

Original: For the edges, we evaluate the effect of having either: a) a binary indicator of bond/contact, b) geometric features, i.e. the euclidean distance between residues, c) sequential features, i.e. the categorical encoding of the separation between residues, d) both distance and separation encoding.
Revised: For the edges, we evaluate the effect of having either: a) a binary indicator of bond/contact, b) geometric features, i.e. the euclidean distance between residues, c) sequential features, i.e. the categorical encoding of the separation between residues, d) both distance and separation encoding.

Original: Progressively richer edge features seem to be beneﬁt LDDT predictions, while little improvement can be seen at the global level.
Revised: Progressively richer edge features seem to be beneﬁt LDDT predictions, while little improvement can be seen at the global level.

Original: 4. V ISUALIZATION AND E XPLAINABILITY
Revised: 4. V ISUALIZATION AND E XPLAINABILITY

Original: The design of G RAPH QA makes it suitable not only for scoring, but also to identify reﬁnement opportunities for computationally-created decoys.
Revised: The design of G RAPH QA makes it suitable not only for scoring, but also to identify reﬁnement opportunities for computationally-created decoys.

Original: Figure 6 shows a decoy that correctly models the native structure of its target for most of the sequence, but one extremity to which both G RAPH QA and LDDT assign low local scores.
Revised: Figure 6 shows a decoy that correctly models the native structure of its target for most of the sequence, but one extremity to which both G RAPH QA and LDDT assign low local scores.

Original: Unlike LDDT, however, G RAPH QA is fully differentiable and the trained model can be used to explain the factors that inﬂuenced a low score and provide useful feedback for computational structure prediction.
Revised: Unlike LDDT, however, G RAPH QA is fully differentiable and the trained model can be used to explain the factors that inﬂuenced a low score and provide useful feedback for computational structure prediction.

Original: A simple approach for explaining predictions of a differentiable function f ( x ) is Sensitivity Analysis (Baehrens et al., 2010; Simonyan et al., 2014), which uses (cid:107)∇ x f (cid:107) to measure how variations in the input affect the output.
Revised: A simple approach for explaining predictions of a differentiable function f ( x ) is Sensitivity Analysis (Baehrens et al., 2010; Simonyan et al., 2014), which uses (cid:107)∇ x f (cid:107) to measure how variations in the input affect the output.

Original: In ﬁgure 6 we consider the scores predicted for two different residues and compute the magnitude of the gradients w.r.t.
Revised: In ﬁgure 6 we consider the scores predicted for two different residues and compute the magnitude of the gradients w.r.t.

Original: the edges of the graph.
Revised: the edges of the graph.

Original: Interestingly, G RAPH QA
Revised: Interestingly, G RAPH QA

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Layers
Revised: Layers

Original: [Figure]
Revised: [Figure]

Original: is able
Revised: is able

Original: Finally, we measure whether the global predictions of G RAPH QA could be used to improve the contact maps used by computational methods to build protein models.
Revised: Finally, we measure whether the global predictions of G RAPH QA could be used to improve the contact maps used by computational methods to build protein models.

Original: If the network has learned a meaningful scoring function for a decoy, then the gradient of the score w.r.t.
Revised: If the network has learned a meaningful scoring function for a decoy, then the gradient of the score w.r.t.

Original: the contact distances should aim in the direction of the native structure.
Revised: the contact distances should aim in the direction of the native structure.

Original: Considering all decoys of all targets in CASP 11, we obtain an average cosine similarity cos ( ∂ G RAPH QA g /∂
Revised: Considering all decoys of all targets in CASP 11, we obtain an average cosine similarity cos ( ∂ G RAPH QA g /∂

Original: d , d decoy − d native ) of 0 .
Revised: d , d decoy − d native ) of 0 .

Original: 14 ± .
Revised: 14 ± .

Original: 08 , which suggests that gradients can be used as a coarse feedback for end-to-end protein model prediction.
Revised: 08 , which suggests that gradients can be used as a coarse feedback for end-to-end protein model prediction.

Original: [Figure]
Revised: [Figure]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: For the ﬁrst time we applied graph convolutional networks to the important problem of protein quality assessment (QA).
Revised: For the ﬁrst time we applied graph convolutional networks to the important problem of protein quality assessment (QA).

Original: Since proteins are naturally represented as graphs, GCN allowed us to collect the individual beneﬁts of the previous QA methods including representation learning, geometric invariance, explicit modeling of sequential and 3D structure, simultaneous local and global scoring, and computational efﬁciency.
Revised: Since proteins are naturally represented as graphs, GCN allowed us to collect the individual beneﬁts of the previous QA methods including representation learning, geometric invariance, explicit modeling of sequential and 3D structure, simultaneous local and global scoring, and computational efﬁciency.

Original: Thanks to these beneﬁts, and through an extensive set of experiments, we demonstrated signiﬁcant improvements upon the state-of-the-art results on various metrics and datasets and further analyzed the results via thorough ablation and qualitative studies.
Revised: Thanks to these beneﬁts, and through an extensive set of experiments, we demonstrated signiﬁcant improvements upon the state-of-the-art results on various metrics and datasets and further analyzed the results via thorough ablation and qualitative studies.

Original: Finally, we wish that Quality Assessment will gain popularity in the machine learning community, that could beneﬁt from several curated datasets and ongoing regular challenges.
Revised: Finally, we wish that Quality Assessment will gain popularity in the machine learning community, that could beneﬁt from several curated datasets and ongoing regular challenges.

Original: We believe that richer geometric representations, e.g. including relative rotations, and raw atomic representations could represent an interesting future direction for learning-based Quality Assessment.
Revised: We believe that richer geometric representations, e.g. including relative rotations, and raw atomic representations could represent an interesting future direction for learning-based Quality Assessment.

Original: A P ROTEIN Q UALITY A SSESSMENT
Revised: A P ROTEIN Q UALITY A SSESSMENT

Original: For the interested reader, we describe here in more detail how the Global Distance Test Total Score (Zemla, 2003) and the Local Distance Difference Test (Mariani et al., 2013) are computed.
Revised: For the interested reader, we describe here in more detail how the Global Distance Test Total Score (Zemla, 2003) and the Local Distance Difference Test (Mariani et al., 2013) are computed.

Original: Furthermore, we provide an intuition over what the beneﬁts and downsides of each method are and motivate why a better quality assessment should consider both a global measure and a local measure.
Revised: Furthermore, we provide an intuition over what the beneﬁts and downsides of each method are and motivate why a better quality assessment should consider both a global measure and a local measure.

Original: Global Distance Test Total Score (GDT TS)
Revised: Global Distance Test Total Score (GDT TS)

Original: Global Distance Test Total Score (GDT TS) is a global-level score obtained by ﬁrst superimposing the structure of a decoy to the experimental structure using an alignment heuristic, and then computing the fraction of residues whose position is within a certain distance from the corresponding residue in the native structure (ﬁgure 7).
Revised: Global Distance Test Total Score (GDT TS) is a global-level score obtained by ﬁrst superimposing the structure of a decoy to the experimental structure using an alignment heuristic, and then computing the fraction of residues whose position is within a certain distance from the corresponding residue in the native structure (ﬁgure 7).

Original: This percentage is computed at different thresholds and then averaged to produce a score in the range [0 , 100] , which we rescale between 0 and 1 (table 2).
Revised: This percentage is computed at different thresholds and then averaged to produce a score in the range [0 , 100] , which we rescale between 0 and 1 (table 2).

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: Local Distance Difference Test (LDDT) Local Distance Difference Test (LDDT), is a residue-level score that does not require alignment of the structures and compares instead the local neighborhood of every residue, in the decoy and in the native structure.
Revised: Local Distance Difference Test (LDDT) Local Distance Difference Test (LDDT), is a residue-level score that does not require alignment of the structures and compares instead the local neighborhood of every residue, in the decoy and in the native structure.

Original: If we deﬁne the neighborhood of a residue as the set of its contacts, i.e. the set of other residues that lie within a certain distance from it, we can express the quality of that residue as the percentage of contacts that it shares with the corresponding residue in the native structure.
Revised: If we deﬁne the neighborhood of a residue as the set of its contacts, i.e. the set of other residues that lie within a certain distance from it, we can express the quality of that residue as the percentage of contacts that it shares with the corresponding residue in the native structure.

Original: 11
Revised: 11

Original: B D ATASETS AND STATISTICS B. D ATASETS
Revised: B D ATASETS AND STATISTICS B. D ATASETS

Original: We consider all decoys of all target included in CASP 7-13, excluding proteins whose sequence is shorter than 50 residues and targets that have been canceled by the organizers.
Revised: We consider all decoys of all target included in CASP 7-13, excluding proteins whose sequence is shorter than 50 residues and targets that have been canceled by the organizers.

Original: In addition to CASP datasets, we test our method on all targets published in CAMEO (Haas et al., 2013) between July and December 2017.
Revised: In addition to CASP datasets, we test our method on all targets published in CAMEO (Haas et al., 2013) between July and December 2017.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: P ROTEIN GRAPHS STATISTICS
Revised: P ROTEIN GRAPHS STATISTICS

Original: The cutoff value d max determines which edges are included in the graph and, consequentially, its connectivity.
Revised: The cutoff value d max determines which edges are included in the graph and, consequentially, its connectivity.

Original: A low cutoff implies a sparsely connected graph, with few edges and long paths between nodes.
Revised: A low cutoff implies a sparsely connected graph, with few edges and long paths between nodes.

Original: A higher cutoff yields a denser graph with more edges and shorter paths.
Revised: A higher cutoff yields a denser graph with more edges and shorter paths.

Original: In ﬁgure 9 we report some statistics about number of edges, average degree and average shortest paths, evaluated at different cutoff values on 1700 decoys from CASP 11.
Revised: In ﬁgure 9 we report some statistics about number of edges, average degree and average shortest paths, evaluated at different cutoff values on 1700 decoys from CASP 11.

Original: [Figure]
Revised: [Figure]

Original: C G RAPH QA ARCHITECTURE
Revised: C G RAPH QA ARCHITECTURE

Original: In this section
Revised: In this section

Original: [Equation]
Revised: [Equation]

Original: we illustrate in more detail the structure of the G RAPH QA architecture, as well as the hyperparameter space that was explored to optimize performances on the validation set.
Revised: we illustrate in more detail the structure of the G RAPH QA architecture, as well as the hyperparameter space that was explored to optimize performances on the validation set.

Original: C. M ESSAGE - PASSING LAYERS
Revised: C. M ESSAGE - PASSING LAYERS

Original: Within G RAPH QA
Revised: Within G RAPH QA

Original: [Equation]
Revised: [Equation]

Original: a protein structure is represented as a graph whose nodes correspond to residues and whose edges connect interacting pairs of amino acids.
Revised: a protein structure is represented as a graph whose nodes correspond to residues and whose edges connect interacting pairs of amino acids.

Original: At the input, the features of the i -th residue are encoded in a node feature vector v i .
Revised: At the input, the features of the i -th residue are encoded in a node feature vector v i .

Original: Similarly, the features of the pairwise interaction between residues i and j are encoded in an edge feature vector e i,j .
Revised: Similarly, the features of the pairwise interaction between residues i and j are encoded in an edge feature vector e i,j .

Original: A global bias term u is also added to represent information that is not localized to any speciﬁc node/edge of the graph.
Revised: A global bias term u is also added to represent information that is not localized to any speciﬁc node/edge of the graph.

Original: With this graph representation
Revised: With this graph representation

Original: [Equation]
Revised: [Equation]

Original: one layer of message passing performs the following updates.
Revised: one layer of message passing performs the following updates.

Original: For every edge i → j , the edge feature vector is updated using a function φ e of adjacent nodes v i and v j , of the edge itself e i,j and of the global attribute u :
Revised: For every edge i → j , the edge feature vector is updated using a function φ e of adjacent nodes v i and v j , of the edge itself e i,j and of the global attribute u :

Original: [Equation]
Revised: [Equation]

Original: For every node i , features from incident edges { e (cid:48)
Revised: For every node i , features from incident edges { e (cid:48)

Original: j,i } are aggregated using a pooling function ρ e →
Revised: j,i } are aggregated using a pooling function ρ e →

Original: [Equation]
Revised: [Equation]

Original: For every node i , the node feature vector is updated using a function φ v of aggregated incident edges ¯ e (cid:48)
Revised: For every node i , the node feature vector is updated using a function φ v of aggregated incident edges ¯ e (cid:48)

Original: i , of the node itself v i and of the global attribute u :
Revised: i , of the node itself v i and of the global attribute u :

Original: [Equation]
Revised: [Equation]

Original: All edges are aggregated using a pooling function ρ
Revised: All edges are aggregated using a pooling function ρ

Original: [Equation]
Revised: [Equation]

Original: u :
Revised: u :

Original: [Equation]
Revised: [Equation]

Original: All nodes are aggregated using a pooling function ρ v → u :
Revised: All nodes are aggregated using a pooling function ρ v → u :

Original: [Equation]
Revised: [Equation]

Original: The global feature vector is updated using a function φ u of the aggregated edges ¯ e (cid:48) , of the aggregated nodes ¯ v (cid:48) and of the global attribute u :
Revised: The global feature vector is updated using a function φ u of the aggregated edges ¯ e (cid:48) , of the aggregated nodes ¯ v (cid:48) and of the global attribute u :

Original: [Equation]
Revised: [Equation]

Original: In G RAPH QA, all intermediate updates are implemented as Linear-Dropout-ReLU functions, and all aggregation functions use average pooling.
Revised: In G RAPH QA, all intermediate updates are implemented as Linear-Dropout-ReLU functions, and all aggregation functions use average pooling.

Original: The encoder and readout layers do not make use of message passing, effectively processing every node/edge in isolation.
Revised: The encoder and readout layers do not make use of message passing, effectively processing every node/edge in isolation.

Original: Message passing is instead enabled for the core layers of the network and enables G RAPH QA to process information within progressively expanding neighborhoods.
Revised: Message passing is instead enabled for the core layers of the network and enables G RAPH QA to process information within progressively expanding neighborhoods.

Original: The number of neurons in the core message-passing layers decreases from the input to the output.
Revised: The number of neurons in the core message-passing layers decreases from the input to the output.

Original: Speciﬁcally it follows a linear interpolation between the input and output numbers reported below, rounded to the closest power of two.
Revised: Speciﬁcally it follows a linear interpolation between the input and output numbers reported below, rounded to the closest power of two.

Original: In preliminary experiments, we noticed that a progressive increase of the number of layers results in convergence issues, which is in contrast to the practice of increasing the number of channels in Convolutional Neural Networks.
Revised: In preliminary experiments, we noticed that a progressive increase of the number of layers results in convergence issues, which is in contrast to the practice of increasing the number of channels in Convolutional Neural Networks.

Original: C. H YPERPARAMETER OPTIMIZATION
Revised: C. H YPERPARAMETER OPTIMIZATION

Original: We perform a guided grid search over the following hyper parameter space.
Revised: We perform a guided grid search over the following hyper parameter space.

Original: The ﬁnal model is chosen to be the one with the highest R target on the validation set.
Revised: The ﬁnal model is chosen to be the one with the highest R target on the validation set.

Original: The following considerations were made:
Revised: The following considerations were made:

Original: • The values for d max are chosen on the base that the typical bond length is ∼ 5 ˚A and residueresidue interactions are negligible after ∼ 10 ˚A. • The values for σ are chosen so that the RBF encoding of the edge length is approximately linear around ∼ 5 ˚A. •
Revised: • The values for d max are chosen on the base that the typical bond length is ∼ 5 ˚A and residueresidue interactions are negligible after ∼ 10 ˚A. • The values for σ are chosen so that the RBF encoding of the edge length is approximately linear around ∼ 5 ˚A. •

Original: The values for L are chosen to approximately match the average length of the shortest paths in the protein graphs at different cutoffs.
Revised: The values for L are chosen to approximately match the average length of the shortest paths in the protein graphs at different cutoffs.

Original: • In addition to what described in section 2.3, we also tested an architecture with BatchNorm layers between the Dropout and ReLU operations, but apart from a signiﬁcant slowdown we did not notice any improvement.
Revised: • In addition to what described in section 2.3, we also tested an architecture with BatchNorm layers between the Dropout and ReLU operations, but apart from a signiﬁcant slowdown we did not notice any improvement.

Original: [Table]
Revised: [Table]

Original: D A DDITIONAL S TUDIES
Revised: D A DDITIONAL S TUDIES

Original: To complement the analysis reported in the main text, we perform additional studies on the effect of feature representation and on the generalization ability of the trained model.
Revised: To complement the analysis reported in the main text, we perform additional studies on the effect of feature representation and on the generalization ability of the trained model.

Original: D. D ISTANCE AND SEPARATION ENCODING
Revised: D. D ISTANCE AND SEPARATION ENCODING

Original: The feature vectors associated to the edge of the graph represent two types of distances between residues, namely spatial distance and separation in the sequence.
Revised: The feature vectors associated to the edge of the graph represent two types of distances between residues, namely spatial distance and separation in the sequence.

Original: In this study we evaluate the effect of different representations on validation performances.
Revised: In this study we evaluate the effect of different representations on validation performances.

Original: Spatial distance is the physical distance between ammino acids, measured as the euclidean distance between their β carbon atoms.
Revised: Spatial distance is the physical distance between ammino acids, measured as the euclidean distance between their β carbon atoms.

Original: We consider three possible encodings for this distance:
Revised: We consider three possible encodings for this distance:

Original: • Absent: spatial distance is not provided as input; • Scalar: spatial distance is provided as a raw scalar value (in ˚Angstrom); • RBF: spatial distance is encoded using 32 RBF kernels, with unit variance, equally spaced between 0 and 20.
Revised: • Absent: spatial distance is not provided as input; • Scalar: spatial distance is provided as a raw scalar value (in ˚Angstrom); • RBF: spatial distance is encoded using 32 RBF kernels, with unit variance, equally spaced between 0 and 20.

Original: Figure 10 reports the aggregated performances on CASP 11 of ten runs for each of the above.
Revised: Figure 10 reports the aggregated performances on CASP 11 of ten runs for each of the above.

Original: The rich representation of the RBF kernels seem to improve both LDDT and GDT TS scoring performances, even though the effect is rather limited.
Revised: The rich representation of the RBF kernels seem to improve both LDDT and GDT TS scoring performances, even though the effect is rather limited.

Original: [Figure]
Revised: [Figure]

Original: Separation is the number of residues between to amino acids in the sequence, we consider three possible encodings:
Revised: Separation is the number of residues between to amino acids in the sequence, we consider three possible encodings:

Original: • Absent: sequential separation is not provided as input; • Scalar: sequential separation is provided as a raw scalar value (positive integer); • Categorical: sequential separation is encoded as a one-hot categorical variable, according to the classes { 0 , 1 , 2 , 3 , 4 , 5 : 10 , > 10 } , which are based on typical interaction patterns within a peptidic chain.
Revised: • Absent: sequential separation is not provided as input; • Scalar: sequential separation is provided as a raw scalar value (positive integer); • Categorical: sequential separation is encoded as a one-hot categorical variable, according to the classes { 0 , 1 , 2 , 3 , 4 , 5 : 10 , > 10 } , which are based on typical interaction patterns within a peptidic chain.

Original: Figure 11 reports the aggregated performances on CASP 11 of ten runs for each of the above.
Revised: Figure 11 reports the aggregated performances on CASP 11 of ten runs for each of the above.

Original: For local scoring, the choice of encoding plays little difference as long as separation is present in the input.
Revised: For local scoring, the choice of encoding plays little difference as long as separation is present in the input.

Original: On the other hand, the choice of categorical encoding over scalar encoding results in higher global scoring performance
Revised: On the other hand, the choice of categorical encoding over scalar encoding results in higher global scoring performance

Original: [Figure]
Revised: [Figure]

Original: D. T RANSMEMBRANE VS . SOLUBLE PROTEINS
Revised: D. T RANSMEMBRANE VS . SOLUBLE PROTEINS

Original: In this study, we evaluate how the natural environment of a protein affects the predictive performances of our method.
Revised: In this study, we evaluate how the natural environment of a protein affects the predictive performances of our method.

Original: Targets from CASP 11 and 12 are classiﬁed as transmembrane and soluble according to Peters et al.
Revised: Targets from CASP 11 and 12 are classiﬁed as transmembrane and soluble according to Peters et al.

Original: and
Revised: and

Original: scored separately using G RAPH QA.
Revised: scored separately using G RAPH QA.

Original: Transmembrane proteins behave differently from soluble proteins as a consequence of the environment they are placed in.
Revised: Transmembrane proteins behave differently from soluble proteins as a consequence of the environment they are placed in.

Original: The former expose non-polar residues to the cellular membrane that surrounds their structure.
Revised: The former expose non-polar residues to the cellular membrane that surrounds their structure.

Original: On the contrary, the latter tend to present polar amino acids to the surrounding water-based solvent.
Revised: On the contrary, the latter tend to present polar amino acids to the surrounding water-based solvent.

Original: Since this information is not explicitly provided to the model, we can compare the predictive performances between the two sets and check that it has actually learned a ﬂexible protein representation.
Revised: Since this information is not explicitly provided to the model, we can compare the predictive performances between the two sets and check that it has actually learned a ﬂexible protein representation.

Original: The outcome of this evaluation is shown in table 5.
Revised: The outcome of this evaluation is shown in table 5.

Original: While it is evident that G RAPH QA performs better on soluble proteins, which are more numerous in the training set, it also scores transmembrane proteins to an acceptable degree.
Revised: While it is evident that G RAPH QA performs better on soluble proteins, which are more numerous in the training set, it also scores transmembrane proteins to an acceptable degree.

Original: [Table]
Revised: [Table]

Original: E A DDITIONAL M ETRICS
Revised: E A DDITIONAL M ETRICS

Original: The Quality Assessment literature is rich of metrics to measure the performances of a scoring method.
Revised: The Quality Assessment literature is rich of metrics to measure the performances of a scoring method.

Original: In the main text we tried to keep the exposition uncluttered by only reporting ﬁgures for the most important metrics.
Revised: In the main text we tried to keep the exposition uncluttered by only reporting ﬁgures for the most important metrics.

Original: Here we present a more extensive set of metrics, that further describe our method and can serve as future benchmark.
Revised: Here we present a more extensive set of metrics, that further describe our method and can serve as future benchmark.

Original: In the following, we use:
Revised: In the following, we use:

Original: [Equation]
Revised: [Equation]

Original: Target proteins
Revised: Target proteins

Original: Decoys of a target
Revised: Decoys of a target

Original: Residue indexes of a target
Revised: Residue indexes of a target

Original: Global quality score (true)
Revised: Global quality score (true)

Original: Local quality scores (true)
Revised: Local quality scores (true)

Original: Global quality score (predicted)
Revised: Global quality score (predicted)

Original: Local quality scores (predicted)
Revised: Local quality scores (predicted)

Original: Root Mean Squared Error (RMSE)
Revised: Root Mean Squared Error (RMSE)

Original: We compute RMSE between all true and predicted scores, for both LDDT and GDT TS.
Revised: We compute RMSE between all true and predicted scores, for both LDDT and GDT TS.

Original: For LDDT, it it the square root of:
Revised: For LDDT, it it the square root of:

Original: [Equation]
Revised: [Equation]

Original: For GDT TS, it it the square root of:
Revised: For GDT TS, it it the square root of:

Original: [Equation]
Revised: [Equation]

Original: Correlation coefﬁcients We compute the Pearson ( R ), Spearman ( ρ ) and Kendall ( τ ) correlation coefﬁcients between all true and predicted scores.
Revised: Correlation coefﬁcients We compute the Pearson ( R ), Spearman ( ρ ) and Kendall ( τ ) correlation coefﬁcients between all true and predicted scores.

Original: Since all scores are treated equally, with no distinction between different decoys or different targets, a high value for these scores can be misleading.
Revised: Since all scores are treated equally, with no distinction between different decoys or different targets, a high value for these scores can be misleading.

Original: Thus, their per-model and per-target versions should be also checked.
Revised: Thus, their per-model and per-target versions should be also checked.

Original: For LDDT:
Revised: For LDDT:

Original: [Equation]
Revised: [Equation]

Original: For GDT TS:
Revised: For GDT TS:

Original: [Equation]
Revised: [Equation]

Original: Correlation coefﬁcients per-model For every decoy of every target, we compute the Pearson ( R model ), Spearman ( ρ model ) and Kendall ( τ model ) correlation coefﬁcients between true and predicted residue-level scores (LDDT).
Revised: Correlation coefﬁcients per-model For every decoy of every target, we compute the Pearson ( R model ), Spearman ( ρ model ) and Kendall ( τ model ) correlation coefﬁcients between true and predicted residue-level scores (LDDT).

Original: We then report the average correlation coefﬁcients across all decoys of all targets.
Revised: We then report the average correlation coefﬁcients across all decoys of all targets.

Original: The per-model correlation coefﬁcients estimate the performance of the network to rank individual residues by their quality and distinguish correctly vs. incorrectly folded segments.
Revised: The per-model correlation coefﬁcients estimate the performance of the network to rank individual residues by their quality and distinguish correctly vs. incorrectly folded segments.

Original: [Equation]
Revised: [Equation]

Original: Correlation coefﬁcients per-target For every target, we compute the Pearson ( R target ), Spearman ( ρ target ) and Kendall ( τ target ) correlation coefﬁcients between true and predicted decoy-level scores (GDT TS).
Revised: Correlation coefﬁcients per-target For every target, we compute the Pearson ( R target ), Spearman ( ρ target ) and Kendall ( τ target ) correlation coefﬁcients between true and predicted decoy-level scores (GDT TS).

Original: We then report the average correlation coefﬁcients across all targets.
Revised: We then report the average correlation coefﬁcients across all targets.

Original: With reference to the funnel plots, this would be the correlation between the markers in every plot, averaged across all plots.
Revised: With reference to the funnel plots, this would be the correlation between the markers in every plot, averaged across all plots.

Original: The per-target correlation coefﬁcients estimate the performance of the network to rank the decoys of a target by their quality and select the ones with highest global quality.
Revised: The per-target correlation coefﬁcients estimate the performance of the network to rank the decoys of a target by their quality and select the ones with highest global quality.

Original: Per-target correlation coefﬁcients are computed only for GDT TS:
Revised: Per-target correlation coefﬁcients are computed only for GDT TS:

Original: [Equation]
Revised: [Equation]

Original: First Rank Loss (FRL)
Revised: First Rank Loss (FRL)

Original: For every target, we compute the difference in GDT TS between the best decoy according to ground-truth scores and best decoy according to the predicted scores.
Revised: For every target, we compute the difference in GDT TS between the best decoy according to ground-truth scores and best decoy according to the predicted scores.

Original: We then report the average FRL across all targets.
Revised: We then report the average FRL across all targets.

Original: This represents the loss in (true) quality we would suffer if we were to choose a decoy according to our rankings and can is represented in the funnel plots by the gap between the two vertical lines indicating the true best (green) and predicted best (red).
Revised: This represents the loss in (true) quality we would suffer if we were to choose a decoy according to our rankings and can is represented in the funnel plots by the gap between the two vertical lines indicating the true best (green) and predicted best (red).

Original: FRL measures the ability to select a single best decoy for a given target.
Revised: FRL measures the ability to select a single best decoy for a given target.

Original: In our experiments, however, we noticed that FRL is extremely subject to noise, as it only considers top-1 decoys.
Revised: In our experiments, however, we noticed that FRL is extremely subject to noise, as it only considers top-1 decoys.

Original: Therefore, we consider NDCG to be a superior metric for this purpose, though we have not seen it used in the QA literature.
Revised: Therefore, we consider NDCG to be a superior metric for this purpose, though we have not seen it used in the QA literature.

Original: FRL is only computed for GDT TS:
Revised: FRL is only computed for GDT TS:

Original: [Equation]
Revised: [Equation]

Original: where d ∗ = arg max d { G RAPH QA g ( P t,d ) } for every target t .
Revised: where d ∗ = arg max d { G RAPH QA g ( P t,d ) } for every target t .

Original: Recall at k (REC @
Revised: Recall at k (REC @

Original: k ) We can describe Quality Assessment as an information retrieval task, where every target represents a query and its decoys are the documents available for retrieval.
Revised: k ) We can describe Quality Assessment as an information retrieval task, where every target represents a query and its decoys are the documents available for retrieval.

Original: If we consider the best decoy to have a score of 1 and all others to have zero score, we can compute the average REC @
Revised: If we consider the best decoy to have a score of 1 and all others to have zero score, we can compute the average REC @

Original: k as the percentage of queries for which the best decoy is retrieved among the top- k results.
Revised: k as the percentage of queries for which the best decoy is retrieved among the top- k results.

Original: This metric, however, is subject to the same pitfalls of FRL, since it only considers the best decoy of every target and ignores the relevance of the others.
Revised: This metric, however, is subject to the same pitfalls of FRL, since it only considers the best decoy of every target and ignores the relevance of the others.

Original: As described below, NDCG offers a better perspective over the decoys retieved by a QA method.
Revised: As described below, NDCG offers a better perspective over the decoys retieved by a QA method.

Original: Normalized Discounted Cumulative Gain at k (NDCG @ k )
Revised: Normalized Discounted Cumulative Gain at k (NDCG @ k )

Original: For a given query we consider the top- k decoys ranked according to their predicted global scores.
Revised: For a given query we consider the top- k decoys ranked according to their predicted global scores.

Original: Discounted Cumulative Gain at k
Revised: Discounted Cumulative Gain at k

Original: (DCG @ k ) is computed as the cumulative sum of their ground-truth GDT TS scores (gain), discounted however according to the position in the list.
Revised: (DCG @ k ) is computed as the cumulative sum of their ground-truth GDT TS scores (gain), discounted however according to the position in the list.

Original: A high DCG @
Revised: A high DCG @

Original: k is obtained therefore by a) selecting the k -best decoys to be part of the top- k predictions, and b) sorting them in order of decreasing quality (the higher in the list, the lower the discount).
Revised: k is obtained therefore by a) selecting the k -best decoys to be part of the top- k predictions, and b) sorting them in order of decreasing quality (the higher in the list, the lower the discount).

Original: Dividing DCG @
Revised: Dividing DCG @

Original: k by DCG ideal @
Revised: k by DCG ideal @

Original: k (obtained by ranking according to the ground-truth scores), yields the Normalized Discounted Cumulative Gain NDCG @
Revised: k (obtained by ranking according to the ground-truth scores), yields the Normalized Discounted Cumulative Gain NDCG @

Original: k ∈ [0 , 1] , which can be compared and averaged across targets.
Revised: k ∈ [0 , 1] , which can be compared and averaged across targets.

Original: F A DDITIONAL R ESULTS
Revised: F A DDITIONAL R ESULTS

Original: In this section we present additional results for datasets, dataset splits, methods, and metrics that are excluded from the main text for sake of brevity.
Revised: In this section we present additional results for datasets, dataset splits, methods, and metrics that are excluded from the main text for sake of brevity.

Original: The datasets considered in the following pages are:
Revised: The datasets considered in the following pages are:

Original: CASP 11, CASP 12, CASP 13, CAMEO.
Revised: CASP 11, CASP 12, CASP 13, CAMEO.

Original: The CASP 11 and 12 datasets are conventionally divided into: stage 1 , containing 20 randomlyselected decoys per target, and stage 2 , containing the top-150 decoys of each target.
Revised: The CASP 11 and 12 datasets are conventionally divided into: stage 1 , containing 20 randomlyselected decoys per target, and stage 2 , containing the top-150 decoys of each target.

Original: In the QA literature, some papers report results either on the dataset as a whole, or on the stage 1 and stagesplits.
Revised: In the QA literature, some papers report results either on the dataset as a whole, or on the stage 1 and stagesplits.

Original: Furthermore, some papers report performances of other methods that differ from the original papers for reasons that are left unspeciﬁed.
Revised: Furthermore, some papers report performances of other methods that differ from the original papers for reasons that are left unspeciﬁed.

Original: In the main text, we adhere to the following rules to summarize the metrics we collected:
Revised: In the main text, we adhere to the following rules to summarize the metrics we collected:

Original: • Metrics computed on stage 1 are considered noisy and ignored, since stage 1 splits contain only 20 randomly-selected decoys per target • Metrics computed on stage 2 and on the whole dataset are considered equally valid, allowing to ”merge” results from papers with different scoring strategies •
Revised: • Metrics computed on stage 1 are considered noisy and ignored, since stage 1 splits contain only 20 randomly-selected decoys per target • Metrics computed on stage 2 and on the whole dataset are considered equally valid, allowing to ”merge” results from papers with different scoring strategies •

Original: If multiple values are reported from multiple sources for the same (method, dataset) pair, only the best one is reported
Revised: If multiple values are reported from multiple sources for the same (method, dataset) pair, only the best one is reported

Original: F. CASP
Revised: F. CASP

Original: [Table]
Revised: [Table]

Original: 1 True
Revised: 1 True

Original: [Figure]
Revised: [Figure]

Original: r e d i c t e d
Revised: r e d i c t e d

Original: R : 0.
Revised: R : 0.

Original: R target : 0.
Revised: R target : 0.

Original: GDT-TS 1 True
Revised: GDT-TS 1 True

Original: [Figure]
Revised: [Figure]

Original: R
Revised: R

Original: [Figure]
Revised: [Figure]

Original: R model
Revised: R model

Original: [Figure]
Revised: [Figure]

Original: LDDT
Revised: LDDT

Original: [Figure]
Revised: [Figure]

Original: v e r a g e
Revised: v e r a g e

Original: R e c a ll
Revised: R e c a ll

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: T T
Revised: T T

Original: T T
Revised: T T

Original: T T
Revised: T T

Original: T T
Revised: T T

Original: T T
Revised: T T

Original: T T
Revised: T T

Original: T T
Revised: T T

Original: T T
Revised: T T

Original: T T
Revised: T T

Original: T T
Revised: T T

Original: T T
Revised: T T

Original: T T
Revised: T T

Original: T T
Revised: T T

Original: T
Revised: T

Original: [Figure]
Revised: [Figure]

Original: T
Revised: T

Original: [Figure]
Revised: [Figure]

Original: T T
Revised: T T

Original: T T
Revised: T T

Original: T T
Revised: T T

Original: T T
Revised: T T

Original: T T
Revised: T T

Original: T T
Revised: T T

Original: T T
Revised: T T

Original: T T
Revised: T T

Original: T T
Revised: T T

Original: T T
Revised: T T

Original: T T
Revised: T T

Original: T T
Revised: T T

Original: T T
Revised: T T

Original: T T
Revised: T T

Original: T T
Revised: T T

Original: T T
Revised: T T

Original: T T
Revised: T T

Original: [Figure]
Revised: [Figure]

Original: F. CASP
Revised: F. CASP

Original: [Table]
Revised: [Table]

Original: 1 True
Revised: 1 True

Original: P r e d i c t e d
Revised: P r e d i c t e d

Original: R : 0.
Revised: R : 0.

Original: R target : 0.
Revised: R target : 0.

Original: GDT-TS 1 True
Revised: GDT-TS 1 True

Original: [Table]
Revised: [Table]

Original: LDDT
Revised: LDDT

Original: [Figure]
Revised: [Figure]

Original: A v e r a g e
Revised: A v e r a g e

Original: R e c a ll @
Revised: R e c a ll @

Original: k
Revised: k

Original: [Figure]
Revised: [Figure]

Original: T
Revised: T

Revised: T

Original: T T 
Revised: T T T

Original: T TT
Revised: T T

Original: T TT
Revised: T T

Original: T TT
Revised: T T

Original: T TT
Revised: T T

Original: T TT
Revised: T T

Original: T TT
Revised: T T

Original: T TT
Revised: T T

Original: T T
Revised: T T

Original: T T
Revised: T T

Original: T T
Revised: T T

Original: T T 
Revised: T T T

Original: T 
Revised: T T T T

Original: T T 
Revised: T T T

Original: T
Revised: T

Original: [Figure]
Revised: [Figure]

Original: P
Revised: P

Original: [Figure]
Revised: [Figure]

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: At a close analysis, it appears that the model completely fails to score decoys of target T060 and defaults to predicting a small constant value.
Revised: At a close analysis, it appears that the model completely fails to score decoys of target T060 and defaults to predicting a small constant value.

Original: To help pinpointing the problem, we compare the predictions made by G RAPH QA and G RAPH QA R AW (available in the repository).
Revised: To help pinpointing the problem, we compare the predictions made by G RAPH QA and G RAPH QA R AW (available in the repository).

Original: It turns out that the model trained on amino acid identity only does not output the same degenerate predictions as its fully-featured counterpart (the predictions are not perfect, but deﬁnitely better than a constant).
Revised: It turns out that the model trained on amino acid identity only does not output the same degenerate predictions as its fully-featured counterpart (the predictions are not perfect, but deﬁnitely better than a constant).

Original: We suspect that an error in the preprocessing pipeline might have produced misleading features for T060, e.g. the multiple sequence alignment program that extracts self information and partial entropy, or the DSSP program that computes secondary structure features.
Revised: We suspect that an error in the preprocessing pipeline might have produced misleading features for T060, e.g. the multiple sequence alignment program that extracts self information and partial entropy, or the DSSP program that computes secondary structure features.

Original: F. CASP
Revised: F. CASP

Original: CASP 13 is the most recent edition of CASP and, at the moment, only few targets are available for public evaluation, i.e. their decoy structures are fully characterized, submitted predictions and ground-truth GDT TS scores can be downloaded from the website.
Revised: CASP 13 is the most recent edition of CASP and, at the moment, only few targets are available for public evaluation, i.e. their decoy structures are fully characterized, submitted predictions and ground-truth GDT TS scores can be downloaded from the website.

Original: Here we present an evaluation on publicly available targets, while waiting to update these results as more data is released.
Revised: Here we present an evaluation on publicly available targets, while waiting to update these results as more data is released.

Original: Also, it is important to note that G RAPH QA is only trained on CASP 7-10, while other participants have likely (re)trained their models on all previous CASP datasets as well as other datasets.
Revised: Also, it is important to note that G RAPH QA is only trained on CASP 7-10, while other participants have likely (re)trained their models on all previous CASP datasets as well as other datasets.

Original: However, even without retraining, we achieve performances that are in line with the results presented for
Revised: However, even without retraining, we achieve performances that are in line with the results presented for

Original: CASP 11 and 12.
Revised: CASP 11 and 12.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: 1 True
Revised: 1 True

Original: P r e d i c t e d
Revised: P r e d i c t e d

Original: R : 0.
Revised: R : 0.

Original: R target : 0.
Revised: R target : 0.

Original: GDT-TS 1 True
Revised: GDT-TS 1 True

Original: [Figure]
Revised: [Figure]

Original: r e d i c t e d
Revised: r e d i c t e d

Original: R 0.
Revised: R 0.

Original: R model 0.
Revised: R model 0.

Original: LDDT
Revised: LDDT

Original: [Figure]
Revised: [Figure]

Original: v
Revised: v

Original: [Figure]
Revised: [Figure]

Original: r a g e
Revised: r a g e

Original: [Figure]
Revised: [Figure]

Original: c a ll
Revised: c a ll

Original: [Figure]
Revised: [Figure]

Original: T T
Revised: T T

Original: T0953s T0953s
Revised: T0953s T0953s

Original: T
Revised: T

Original: T0957s T0957s
Revised: T0957s T0957s

Original: T
Revised: T

Original: T T
Revised: T T

Original: T
Revised: T

Original: T0968s T0968s
Revised: T0968s T0968s

Original: T
Revised: T

Original: T T
Revised: T T

Original: T T
Revised: T T

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: P r e d i c t e d
Revised: P r e d i c t e d

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 1 True
Revised: 1 True

Original: [Figure]
Revised: [Figure]

Original: R
Revised: R

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: GDT-TS 1 True
Revised: GDT-TS 1 True

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: R
Revised: R

Original: [Figure]
Revised: [Figure]

Original: R
Revised: R

Original: [Figure]
Revised: [Figure]

Original: LDDT
Revised: LDDT

Original: [Figure]
Revised: [Figure]

Original: 2ML1_A 2ML2_A 2ML3_A 2MNT_A
Revised: 2ML1_A 2ML2_A 2ML3_A 2MNT_A

Original: 2MP6_A 2MP8_A 2MPH_A 2MPK_A 2MPW_A 2MQ8_A 2MQB_A 2MQC_A 2MQD_A 2MQH_A 2MQM_A
Revised: 2MP6_A 2MP8_A 2MPH_A 2MPK_A 2MPW_A 2MQ8_A 2MQB_A 2MQC_A 2MQD_A 2MQH_A 2MQM_A

Original: 2MR6_A 2MRC_A 2MRL_A
Revised: 2MR6_A 2MRC_A 2MRL_A

Original: 2MSV_A 2MU0_A 2MUI_A 2MVB_A 2MW1_A 2MXD_A 2MXX_A 2MZZ_A 2N00_A 2N03_A 2N0P_A 2N1D_A 2RTT_A 3WCG_D 3WD6_D 3WDX_B 3WE5_B 3WE9_A 3WFX_B 3WGQ_B 3WH9_B 3WJ1_A 3WJ2_D 3WJO_B 1 True
Revised: 2MSV_A 2MU0_A 2MUI_A 2MVB_A 2MW1_A 2MXD_A 2MXX_A 2MZZ_A 2N00_A 2N03_A 2N0P_A 2N1D_A 2RTT_A 3WCG_D 3WD6_D 3WDX_B 3WE5_B 3WE9_A 3WFX_B 3WGQ_B 3WH9_B 3WJ1_A 3WJ2_D 3WJO_B 1 True

Original: [Figure]
Revised: [Figure]

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: d
Revised: d

Original: [Figure]
Revised: [Figure]

Original: 3WTB_H 3WTC_B 3WV6_B
Revised: 3WTB_H 3WTC_B 3WV6_B

Original: 3WV9_D 3WVA_B 3WW1_B 3WWV_A 3WX5_B 3WXV_A 3WXW_A
Revised: 3WV9_D 3WVA_B 3WW1_B 3WWV_A 3WX5_B 3WXV_A 3WXW_A

Original: 3WXW_H 3WY7_D 3WYB_B 3WYD_B 3WYH_B 3X1B_B 3X2E_D 4BWH_A 4CG4_F 4CIT_A 4CJX_B 4CL2_A 4CNE_B 4CNF_B
Revised: 3WXW_H 3WY7_D 3WYB_B 3WYD_B 3WYH_B 3X1B_B 3X2E_D 4BWH_A 4CG4_F 4CIT_A 4CJX_B 4CL2_A 4CNE_B 4CNF_B

Original: 4CNJ_D
Revised: 4CNJ_D

Original: 4CRQ_B 4CRR_A 4CSI_B 4CUK_D
Revised: 4CRQ_B 4CRR_A 4CSI_B 4CUK_D

Original: 4CUO_A 4CVH_A 4CVK_A 4CW9_B 4CWC_C 4CWE_C 4CX8_B 4D2K_D 4D40_B 4D4Z_B 4D6V_A
Revised: 4CUO_A 4CVH_A 4CVK_A 4CW9_B 4CWC_C 4CWE_C 4CX8_B 4D2K_D 4D40_B 4D4Z_B 4D6V_A

Original: 4D79_D 4D7S_B
Revised: 4D79_D 4D7S_B

Original: [Figure]
Revised: [Figure]

Original: 4LM6_D
Revised: 4LM6_D

Original: 4LMS_A 4LMS_C
Revised: 4LMS_A 4LMS_C

Original: 4LMX_C 4LMX_K 4LMX_L 4LPL_A 4LQR_A 4LUG_B 4LUR_A 4LVC_D
Revised: 4LMX_C 4LMX_K 4LMX_L 4LPL_A 4LQR_A 4LUG_B 4LUR_A 4LVC_D

Original: 4LWL_A 4LX4_D 4LZL_A 4M0X_B 4M1X_D 4M3K_B 4MAG_A 4MFI_A 4MH1_B 4MHL_A
Revised: 4LWL_A 4LX4_D 4LZL_A 4M0X_B 4M1X_D 4M3K_B 4MAG_A 4MFI_A 4MH1_B 4MHL_A

Original: 4MHU_B 4MMO_B
Revised: 4MHU_B 4MMO_B

Original: 4MMP_A 4MNP_A 4MTR_B 4MXG_D 4MYI_A 4MYJ_A 4N1H_D 4N1L_A 4N1V_A 4N81_A
Revised: 4MMP_A 4MNP_A 4MTR_B 4MXG_D 4MYI_A 4MYJ_A 4N1H_D 4N1L_A 4N1V_A 4N81_A

Original: 4NCP_F 4NEX_B 4NF7_A
Revised: 4NCP_F 4NEX_B 4NF7_A

Original: 4NJ8_B 4NKN_F 4NKR_E 4NL9_B
Revised: 4NJ8_B 4NKN_F 4NKR_E 4NL9_B

Original: 4NL9_D 4NOV_A 

Revised: 4NL9_D 4NOV_A

Original: True
Revised: True

Original: [Figure]
Revised: [Figure]

Original: d 4NR
Revised: d 4NR

Original: [Figure]
Revised: [Figure]

Original: PredictedTrue
Revised: PredictedTrue

Original: [Figure]
Revised: [Figure]

Original: 4O8V_B 4O9H_L 4OB1_A 4OB7_A
Revised: 4O8V_B 4O9H_L 4OB1_A 4OB7_A

Original: 4OD6_A
Revised: 4OD6_A

Original: 4OGD_B 4OIB_A 4OJU_D 4OJX_A 4OKZ_D 4OLO_D 4OLP_D 4ONY_B 4ORA_A
Revised: 4OGD_B 4OIB_A 4OJU_D 4OJX_A 4OKZ_D 4OLO_D 4OLP_D 4ONY_B 4ORA_A

Original: 4OSP_D 4OTP_A
Revised: 4OSP_D 4OTP_A

Original: 4OUA_B 4P29_B 4PAC_A 4PDE_A 4PEN_A
Revised: 4OUA_B 4P29_B 4PAC_A 4PDE_A 4PEN_A

Original: 4PKM_A 4PLP_B 4PNF_H
Revised: 4PKM_A 4PLP_B 4PNF_H

Original: 4PNG_B 4PQH_B 4PQI_A 4PQX_D 4PRS_B
Revised: 4PNG_B 4PQH_B 4PQI_A 4PQX_D 4PRS_B

Original: 4PSD_A 4PTZ_D 4PU3_B 4PU7_A 4PUG_B
Revised: 4PSD_A 4PTZ_D 4PU3_B 4PU7_A 4PUG_B

Original: 4PUI_B 4PUX_A 4PV3_C 4PV3_D 4PW1_B
Revised: 4PUI_B 4PUX_A 4PV3_C 4PV3_D 4PW1_B

Original: 4PW3_D 4PW9_B 4PWU_D 1 True
Revised: 4PW3_D 4PW9_B 4PWU_D 1 True

Original: P r e d i c t e d
Revised: P r e d i c t e d

Original: [Figure]
Revised: [Figure]

Original: PredictedTrue
Revised: PredictedTrue

Original: [Figure]
Revised: [Figure]

Original: C
Revised: C

Original: [Figure]
Revised: [Figure]

Original: D
Revised: D

Original: [Figure]
Revised: [Figure]

Original: C
Revised: C

Original: [Figure]
Revised: [Figure]

Original: 4Q32_D 4Q34_A 4Q3U_D 4Q47_B 4Q4T_A 4Q53_B 4Q5T_A
Revised: 4Q32_D 4Q34_A 4Q3U_D 4Q47_B 4Q4T_A 4Q53_B 4Q5T_A

Original: 4Q69_B 4Q6K_D 4Q6U_B 4Q76_B 4Q7A_D 4Q97_B 4Q9A_B 4Q9B_B 4Q9C_A 4Q9D_B 4Q9O_B 4QAF_B 4QAN_B 4QAW_H
Revised: 4Q69_B 4Q6K_D 4Q6U_B 4Q76_B 4Q7A_D 4Q97_B 4Q9A_B 4Q9B_B 4Q9C_A 4Q9D_B 4Q9O_B 4QAF_B 4QAN_B 4QAW_H

Original: 4QB1_A 4QB5_D
Revised: 4QB1_A 4QB5_D

Original: 4QB7_A 4QBA_B 4QBD_C
Revised: 4QB7_A 4QBA_B 4QBD_C

Original: 4QBD_D
Revised: 4QBD_D

Original: 4QCC_B
Revised: 4QCC_B

Original: 4QD8_F 4QDG_B 4QDY_A 4QE0_B 4QEA_G 4QEY_J 4QEZ_C 4QF3_B 4QFH_B 4QFU_L 4QG5_D 4QGM_A 4QGN_A 4QGR_B 

Revised: 4QD8_F 4QDG_B 4QDY_A 4QE0_B 4QEA_G 4QEY_J 4QEZ_C 4QF3_B 4QFH_B 4QFU_L 4QG5_D 4QGM_A 4QGN_A 4QGR_B

Original: True
Revised: True

Original: [Figure]
Revised: [Figure]

Original: d
Revised: d

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 4QL5_
Revised: 4QL5_

Original: [Figure]
Revised: [Figure]

Original: 4QLO_A
Revised: 4QLO_A

Original: 4QMF_B 4QMG_E
Revised: 4QMF_B 4QMG_E

Original: 4QN8_C 4QNI_A 4QNS_D 4QNU_H 4QNY_B 4QOA_A
Revised: 4QN8_C 4QNI_A 4QNS_D 4QNU_H 4QNY_B 4QOA_A

Original: 4QOX_A
Revised: 4QOX_A

Original: 4QPD_B 4QPG_C 4QPI_A 4QPI_B 4QPO_D 4QPV_B 4QPY_C 4QQ7_B 4QRI_B 4QRJ_B 4QRK_A 4QRL_A
Revised: 4QPD_B 4QPG_C 4QPI_A 4QPI_B 4QPO_D 4QPV_B 4QPY_C 4QQ7_B 4QRI_B 4QRJ_B 4QRK_A 4QRL_A

Original: 4QRZ_A 4QS4_A 4QT8_A 4QT9_B 4QTP_D 4QTZ_A 4QU7_D 4QVG_D
Revised: 4QRZ_A 4QS4_A 4QT8_A 4QT9_B 4QTP_D 4QTZ_A 4QU7_D 4QVG_D

Original: 4QVK_B
Revised: 4QVK_B

Original: 4QVR_A 4QVS_A 4QVU_A 4QWO_B 4QX6_D 4QYO_A
Revised: 4QVR_A 4QVS_A 4QVU_A 4QWO_B 4QX6_D 4QYO_A

Original: 4QYR_A 4R01_B 4R03_A 4R0C_D 4R0G_D 1 True
Revised: 4QYR_A 4R01_B 4R03_A 4R0C_D 4R0G_D 1 True

Original: P r e d i c t e d 4R0J
Revised: P r e d i c t e d 4R0J

Original: [Figure]
Revised: [Figure]

Original: PredictedTrue
Revised: PredictedTrue

Original: [Figure]
Revised: [Figure]

Original: A
Revised: A

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: 4R2B_B 4R2F_A
Revised: 4R2B_B 4R2F_A

Original: 4R31_F 4R33_B 4R37_B 4R4B_F 4R4G_A 4R4H_A 4R4N_V 4R5O_D 4R60_B 4R6H_A 4R6K_A 4R6Y_A 4R7E_A 4R7F_A 4R7K_D 4R7O_G 4R7Q_A 4R7R_A 4R7S_A 4R7T_C 4R80_B 4R86_B
Revised: 4R31_F 4R33_B 4R37_B 4R4B_F 4R4G_A 4R4H_A 4R4N_V 4R5O_D 4R60_B 4R6H_A 4R6K_A 4R6Y_A 4R7E_A 4R7F_A 4R7K_D 4R7O_G 4R7Q_A 4R7R_A 4R7S_A 4R7T_C 4R80_B 4R86_B

Original: 4R8O_B 4R99_D 4R9M_C 4R9N_A 4R9X_B 4R9Z_B 4RAA_A 4RAS_C
Revised: 4R8O_B 4R99_D 4R9M_C 4R9N_A 4R9X_B 4R9Z_B 4RAA_A 4RAS_C

Original: 4RBM_A
Revised: 4RBM_A

Original: 4RC1_I
Revised: 4RC1_I

Original: 4RCK_B
Revised: 4RCK_B

Original: 4RCO_B 4RD7_A 4RD8_B 4RDB_A
Revised: 4RCO_B 4RD7_A 4RD8_B 4RDB_A

Original: 4REI_A 4REV_B 4REW_B
Revised: 4REI_A 4REV_B 4REW_B

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: r
Revised: r

Original: [Figure]
Revised: [Figure]

Original: d i c t
Revised: d i c t

Original: [Figure]
Revised: [Figure]

Original: d
Revised: d

Original: [Figure]
Revised: [Figure]

Original: 4RHC_L 4RHL_B 4RHO_B
Revised: 4RHC_L 4RHL_B 4RHO_B

Original: 4RHT_D 4RK2_B
Revised: 4RHT_D 4RK2_B

Original: 4RK5_A 4RK9_B 4RKQ_D 4RL6_B 4RL9_B 4RLB_B 4RLC_A 4RM7_A 4RMM_A 4RN3_B 4RN7_A 4RNF_A
Revised: 4RK5_A 4RK9_B 4RKQ_D 4RL6_B 4RL9_B 4RLB_B 4RLC_A 4RM7_A 4RMM_A 4RN3_B 4RN7_A 4RNF_A

Original: 4RNJ_B 4RNL_D 4RO3_B 4RO6_A
Revised: 4RNJ_B 4RNL_D 4RO3_B 4RO6_A

Original: 4ROP_A 4ROQ_A 4ROR_A 4ROT_A 4RP5_B 4RPC_B 4RPF_C
Revised: 4ROP_A 4ROQ_A 4ROR_A 4ROT_A 4RP5_B 4RPC_B 4RPF_C

Original: 4RPO_D
Revised: 4RPO_D

Original: 4RQA_A 4RQO_B 4RQQ_D 4RQT_A 4RRP_F 4RS2_B 4RS3_A 4RSH_C 4RSL_A
Revised: 4RQA_A 4RQO_B 4RQQ_D 4RQT_A 4RRP_F 4RS2_B 4RS3_A 4RSH_C 4RSL_A

Original: 4RT5_B 4RTF_D 4RU0_B 4RU1_L 1 True
Revised: 4RT5_B 4RTF_D 4RU0_B 4RU1_L 1 True

Original: P r e d i c t e d 4RUW_A Best decoy
Revised: P r e d i c t e d 4RUW_A Best decoy

Original: PredictedTrue 4RV1_F 4RWE_A 4RWF_A
Revised: PredictedTrue 4RV1_F 4RWE_A 4RWF_A

Original: 4RWG_C 4RWH_A
Revised: 4RWG_C 4RWH_A

Original: 4RWR_B
Revised: 4RWR_B

Original: 4RXL_A 4RXM_B 4RXT_A 4RXU_A 4RY0_A 4RY1_B 4RY8_D 4RY9_B 4RYA_A 4RYB_B 4RYK_A 4RZP_B
Revised: 4RXL_A 4RXM_B 4RXT_A 4RXU_A 4RY0_A 4RY1_B 4RY8_D 4RY9_B 4RYA_A 4RYB_B 4RYK_A 4RZP_B

Original: 4S13_H 4S1R_H 4S1S_H 4S2T_Q 4TPU_A 4TPV_B 4TQJ_B
Revised: 4S13_H 4S1R_H 4S1S_H 4S2T_Q 4TPU_A 4TPV_B 4TQJ_B

Original: 4TQX_A 4TRT_B 4TZH_B 4U00_A 4U06_A 4U08_B 4U09_B 4U7A_A 4U9W_D 4UAO_B
Revised: 4TQX_A 4TRT_B 4TZH_B 4U00_A 4U06_A 4U08_B 4U09_B 4U7A_A 4U9W_D 4UAO_B

Original: 4UDK_F
Revised: 4UDK_F

Original: 4UDP_B 4UE0_C 4UEG_B 4ULV_B 4UMK_D 4UMS_A 4UOH_C 4UOX_D
Revised: 4UDP_B 4UE0_C 4UEG_B 4ULV_B 4UMK_D 4UMS_A 4UOH_C 4UOX_D

Original: 4URJ_D 4URP_B 4USS_A 1 True
Revised: 4URJ_D 4URP_B 4USS_A 1 True

Original: P r e d i c t e d 4UU4_A Best decoy
Revised: P r e d i c t e d 4UU4_A Best decoy

Original: PredictedTrue 4UUC_A
Revised: PredictedTrue 4UUC_A

Original: 4UVQ_A 4UWW_A
Revised: 4UVQ_A 4UWW_A

Original: 4UX7_B 4UXJ_H
Revised: 4UX7_B 4UXJ_H

Original: 4UY4_B 4UY5_A 4UYB_A 4UYI_A 4UYP_D 4UYQ_A 4UYQ_B
Revised: 4UY4_B 4UY5_A 4UYB_A 4UYI_A 4UYP_D 4UYQ_A 4UYQ_B

Original: 4UZ8_B 4UZG_A
Revised: 4UZ8_B 4UZG_A

Original: 4UZM_A
Revised: 4UZM_A

Original: 4UZZ_A 4UZZ_B 4V06_B 4V0P_A
Revised: 4UZZ_A 4UZZ_B 4V06_B 4V0P_A

Original: 4V1G_C 4V2A_A 4V2B_B 4V2E_B 4V2X_A 4V3H_B 4V3I_A 4W8L_C 4WA0_A 4WD1_A 4WGX_D 4WHB_H
Revised: 4V1G_C 4V2A_A 4V2B_B 4V2E_B 4V2X_A 4V3H_B 4V3I_A 4W8L_C 4WA0_A 4WD1_A 4WGX_D 4WHB_H

Original: 4WHN_D 4WIP_C
Revised: 4WHN_D 4WIP_C

Original: 4WMF_C 4WOE_B 4WPZ_A 4WRI_A 4WRP_B 4WSZ_B 4WXJ_B
Revised: 4WMF_C 4WOE_B 4WPZ_A 4WRI_A 4WRP_B 4WSZ_B 4WXJ_B

Original: 4WZX_A 4X0K_A
Revised: 4WZX_A 4X0K_A

Original: 4X36_A 4X3H_A 4X3X_A
Revised: 4X36_A 4X3H_A 4X3X_A

Original: 4X54_A 4X5S_B 1 True
Revised: 4X54_A 4X5S_B 1 True

Original: [Figure]
Revised: [Figure]

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: 4XDZ_B 4XEA_A
Revised: 4XDZ_B 4XEA_A

Original: 4XED_A 4XEL_B 4XEQ_D 4XEU_A 4XFE_A 4XFK_A 4XFR_B 4XFV_A 4XGJ_A
Revised: 4XED_A 4XEL_B 4XEQ_D 4XEU_A 4XFE_A 4XFK_A 4XFR_B 4XFV_A 4XGJ_A

Original: 4XGL_A 4XGU_F 4XHC_B
Revised: 4XGL_A 4XGU_F 4XHC_B

Original: 4XHM_B 4XI2_A
Revised: 4XHM_B 4XI2_A

Original: 4XI5_D 4XIJ_A 4XIN_B
Revised: 4XI5_D 4XIJ_A 4XIN_B

Original: 4XJ6_A 4XK1_B 4XK2_C 4XKZ_A 4XL8_C 4XLL_B 4XLT_A 4XOX_B
Revised: 4XJ6_A 4XK1_B 4XK2_C 4XKZ_A 4XL8_C 4XLL_B 4XLT_A 4XOX_B

Original: 4XP7_A 4XPO_A
Revised: 4XP7_A 4XPO_A

Original: 4XQ1_A 4XQ3_G 4XQ7_A
Revised: 4XQ1_A 4XQ3_G 4XQ7_A

Original: 4XRP_D
Revised: 4XRP_D

Original: 4XRP_E 4XRP_F 4XS5_D 4XTT_B 4XUP_F
Revised: 4XRP_E 4XRP_F 4XS5_D 4XTT_B 4XUP_F

Original: 4XUT_C 4XUV_B 4XV0_A 4XVH_A 1 True
Revised: 4XUT_C 4XUV_B 4XV0_A 4XVH_A 1 True

Original: [Figure]
Revised: [Figure]

Original: 4XXP_A
Revised: 4XXP_A

Original: 4XXT_A 4XYK_D 4Y0B_B 4Y0C_A 4Y61_B 4Y7D_B 4Y85_C 4Y8W_C
Revised: 4XXT_A 4XYK_D 4Y0B_B 4Y0C_A 4Y61_B 4Y7D_B 4Y85_C 4Y8W_C

Original: 4Y97_G 4Y97_H 4Y9J_B 4Y9T_A
Revised: 4Y97_G 4Y97_H 4Y9J_B 4Y9T_A

Original: 4YAY_A
Revised: 4YAY_A

Original: 4YBG_A 4YC6_G 4YC7_B 4YCS_F
Revised: 4YBG_A 4YC6_G 4YC7_B 4YCS_F

Original: 4YDH_C 4YDI_L
Revised: 4YDH_C 4YDI_L

Original: 4YDJ_A 4YDJ_B 4YDK_H 4YDK_L 4YDL_B 4YE2_B 4YE5_B 4YEP_B
Revised: 4YDJ_A 4YDJ_B 4YDK_H 4YDK_L 4YDL_B 4YE2_B 4YE5_B 4YEP_B

Original: 4YEQ_U 4YER_B 4YF1_D 4YFE_B
Revised: 4YEQ_U 4YER_B 4YF1_D 4YFE_B

Original: 4YFG_A 4YFL_F 4YFY_B 4YGQ_A 4YGT_A 4YGU_D
Revised: 4YFG_A 4YFL_F 4YFY_B 4YGQ_A 4YGT_A 4YGU_D

Original: 4YH7_A 4YH7_B 4YHE_B
Revised: 4YH7_A 4YH7_B 4YHE_B

Original: 4YHS_A
Revised: 4YHS_A

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: 4YKC_A
Revised: 4YKC_A

Original: 4YKE_B 4YL5_A 4YLE_A 4YLT_A 4YME_A 4YMR_B 4YMW_A 4YMW_C
Revised: 4YKE_B 4YL5_A 4YLE_A 4YLT_A 4YME_A 4YMR_B 4YMW_A 4YMW_C

Original: 4YMX_B 4YN1_A 4YN2_A 4YNA_D 4YNW_B 4YO7_A 4YOD_A 4YOK_A 4YOO_A
Revised: 4YMX_B 4YN1_A 4YN2_A 4YNA_D 4YNW_B 4YO7_A 4YOD_A 4YOK_A 4YOO_A

Original: 4YPJ_B
Revised: 4YPJ_B

Original: 4YQY_B 4YQZ_D 4YS6_A
Revised: 4YQY_B 4YQZ_D 4YS6_A

Original: 4YS8_D 4YUB_B
Revised: 4YS8_D 4YUB_B

Original: 4YV7_A 4YVD_A 4YWE_H 4YWH_B 4YWJ_B 4YWN_B 4YX1_B 4YX5_A 4YXC_B
Revised: 4YV7_A 4YVD_A 4YWE_H 4YWH_B 4YWJ_B 4YWN_B 4YX1_B 4YX5_A 4YXC_B

Original: 4YY8_B 4YYC_A 4YYF_C 4YZ8_A 4YZH_A 4YZS_B 4YZY_A 4Z04_A 4Z0N_A
Revised: 4YY8_B 4YYC_A 4YYF_C 4YZ8_A 4YZH_A 4YZS_B 4YZY_A 4Z04_A 4Z0N_A

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: d
Revised: d

Original: [Figure]
Revised: [Figure]

Original: 4Z7X_B 4Z8X_C 4Z8Z_A 4ZA2_D 4ZBG_A 4ZCH_B 4ZFN_B
Revised: 4Z7X_B 4Z8X_C 4Z8Z_A 4ZA2_D 4ZBG_A 4ZCH_B 4ZFN_B

Original: 4ZG1_F 4ZGF_A
Revised: 4ZG1_F 4ZGF_A

Original: 4ZHB_A 4ZJF_D
Revised: 4ZHB_A 4ZJF_D

Original: 4ZJM_A 4ZJN_C 4ZJP_A 4ZJS_E 4ZJU_A 4ZN6_B 4ZNM_B 4ZO4_D 4ZPJ_A 4ZQB_B
Revised: 4ZJM_A 4ZJN_C 4ZJP_A 4ZJS_E 4ZJU_A 4ZN6_B 4ZNM_B 4ZO4_D 4ZPJ_A 4ZQB_B

Original: 4ZR7_D 4ZR8_B 4ZRX_A
Revised: 4ZR7_D 4ZR8_B 4ZRX_A

Original: 4ZTI_B 4ZTK_A 4ZV3_C 5A0G_F 5A0L_B 5A0N_A 5A1F_A 5A2L_H 5AF7_B
Revised: 4ZTI_B 4ZTK_A 4ZV3_C 5A0G_F 5A0L_B 5A0N_A 5A1F_A 5A2L_H 5AF7_B

Original: 5AFF_C
Revised: 5AFF_C

Original: 5AGA_A 5AGQ_A 5AH2_D 5AIE_A 5AJ8_B
Revised: 5AGA_A 5AGQ_A 5AH2_D 5AIE_A 5AJ8_B

Original: 5AJA_B 5AJT_D
Revised: 5AJA_B 5AJT_D

Original: 5AL7_B
Revised: 5AL7_B

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: i c
Revised: i c

Original: [Figure]
Revised: [Figure]

Original: d
Revised: d

Original: [Figure] [Figure]
Revised: [Figure] [Figure]