Revised: Under review as a conference paper at ICLR

Original: L IGHTWEIGHT E QUIVARIANT G RAPH R EPRESENTATION L EARNING FOR P ROTEIN E NGINEERING
Revised: L IGHTWEIGHT E QUIVARIANT G RAPH R EPRESENTATION L EARNING FOR P ROTEIN E NGINEERING

Original: Anonymous authors Paper under double-blind review
Revised: Anonymous authors Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: This work tackles the issue of directed evolution in computational protein design that makes accurate predictions of the function of a protein mutant.
Revised: This work tackles the issue of directed evolution in computational protein design that makes accurate predictions of the function of a protein mutant.

Original: We design a lightweight pre-training graph neural network model for multi-task protein representation learning from its 3D structure.
Revised: We design a lightweight zero-shot graph neural network model for multi-task protein representation learning from its 3D structure.

Original: Rather than reconstructing and optimizing the protein structure, the trained model recovers the amino acid types and key properties of the central residues from a given noisy three-dimensional local environment.
Revised: Rather than reconstructing and optimizing the protein structure, the trained model recovers the amino acid types and key properties of the central residues from a given noisy three-dimensional local environment.

Original: On the prediction task for the higher-order mutants, where many amino acid sites of the protein are mutated, the proposedtraining strategy achieves remarkably higher performance by 20% improvement at the cost of requiring less than 1% of computational resources that are required by popular transformer-based state-of-the-art deep learning models for protein design
Revised: On the prediction of higher-order mutations where multiple amino acid sites of the protein are mutated simultaneously, the proposed strategy achieves remarkably higher performance by 20% improvement at the cost of requiring less than 1% of computational resources that are required by popular transformer-based state-of-the-art deep learning models for protein design.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Mutation is a biological process where the amino acid (AA) type of one or multiple sites of a speciﬁc protein is changed.
Revised: Mutation is a biological process where the amino acid (AA) type of one or multiple sites of a speciﬁc protein is changed.

Original: While the wild-type proteins’ functions do not always meet the demand of bio-engineering, it is vital to manually optimize the functionality, namely ﬁtness, with favorable mutations so that they are applicable in designing antibodies (Wu et al., 2019; Pinheiro et al., 2021;
Revised: While the wild-type proteins’ functions do not always meet the demand of bio-engineering, it is vital to manually optimize the functionality, namely ﬁtness, with favorable mutations so that they are applicable in designing antibodies (Wu et al., 2019; Pinheiro et al., 2021;

Original: Shan et al., 2022) or enzymes (Sato & Ishida, 2019; Wittmann et al., 2021).
Revised: Shan et al., 2022) or enzymes (Sato & Ishida, 2019; Wittmann et al., 2021).

Original: A protein usually constitutes hundreds to thousands of AAs, where each residue belongs to one of twenty AA types.
Revised: A protein usually constitutes hundreds to thousands of AAs, where each residue belongs to one of twenty AA types.

Original: To optimize a protein’s functional ﬁtness, a greedy search  in the local sequence is usually conducted, where AA sites were mutated to proper AA types to render a protein mutant with the highest gain-of-function (Rocklin et al., 2017).
Revised: To optimize a protein’s functional ﬁtness, a greedy search is usually conducted in the local sequence, where AA sites are mutated to proper AA types to render a protein mutant with the highest gain-of-function (Rocklin et al., 2017).

Original: Such a process is called directed evolution.
Revised: Such a process is called directed evolution

Original: Normally, multiple AA sites ( ⇠ 5-10) of the protein need to be mutated to obtain a mutant with great ﬁtness (Arnold, 1998; Cui et al., 2021), namely deep mutants.
Revised: Arnold (1998). To obtain a mutant with great ﬁtness, multiple AA sites ( ⇠ 5-10) of the protein need to be mutated, namely deep mutations (see Figure 1).

Original: It requires enormous experimental cost, as the total number of potential combinations of mutations for deep mutants is astronomical.
Revised: It, however, requires enormous experimental costs, as the total number of potential combinations of mutations for deep mutants is astronomical.

Original: Since it is impossible to conduct systematic experimental tests on all possible deep mutants, in silico examination of protein variants’ ﬁtness becomes highly desirable.
Revised: Since it is impossible to conduct systematic experimental tests on all possible deep mutations, in silico examination of protein variants’ ﬁtness becomes highly desirable.

Original: A handful of deep learning methods have been developed to accelerate the discovery of advantageous mutants.
Revised: A handful of deep learning methods have been developed to accelerate the discovery of advantageous mutants.

Original: For instance, 
Revised: For instance, Lu et al.

Original: Sato & Ishida (2019) applied 3DCNN to identify a new polymerase with advantageous single-site mutation and enhanced the speed of degrading PET by 7-8 times at 50  . Luo et al.
Revised: (2022) applied 3DCNN to identify a new polymerase with advantageous single-site mutation and enhanced the speed of degrading PET, i.e., a type of solid waste, by 7-8 times at 50  C. Luo et al.

Original: (2021) proposed ECN ET to use evolutionary context to predict functional ﬁtness for protein engineering, guide the engineering of TEM-1  -lactamase and identify variants with improved ampicillin resistance.
Revised: (2021) proposed ECN ET that predicts functional ﬁtness for protein engineering with evolutionary context. The model guides the engineering of TEM-1  -lactamase and identiﬁes variants with improved ampicillin resistance.

Original: Thean et al.
Revised: Thean et al.

Original: combined deep neural networks and SVM to predict  variants’ activities in multi-sitesaturated mutagenesis libraries froma small sample of variants and identifed Cas9 nuclease variants that possess higher editing activity of derived base editors in human cells.
Revised: (2022) enhanced SVD with deep learning to predict nuclease variants’ activities in multi-site-saturated mutagenesis libraries from and identiﬁed Cas nuclease variants that possess higher editing activity of derived base editors in human cells.

Original: Due to the scarcity of labeled protein data, researchers often pre-train an encoder for unsupervised learning with protein sequences or structures, and use the learned protein representations to train downstream tasks, such as de novo protein design (Hsu et al., 2022), mutation effect prediction (Ingraham et al., 2019;
Revised: Due to the scarcity of labeled protein data, researchers often pre-train an encoder for unsupervised learning with protein sequences or structures, and use the learned protein representations to train speciﬁc tasks, such as de novo protein design (Hsu et al., 2022), mutation effect prediction (Ingraham et al., 2019;

Original: Jing et al., 2020; Meier et al., 2021; Notin et al., 2022), and higher-level structure prediction (Elnaggar et al., 2021).
Revised: Jing et al., 2020; Meier et al., 2021; Notin et al., 2022), and higher-level structure prediction (Elnaggar et al., 2021).

Original: In the context of ﬁtness prediction of mutation effect, existing methods usually transform the problem to mini- de novo design, i.e., infer a speciﬁc AA type from its micro-environment, including the neighborhood AA types.
Revised: In the context of ﬁtness prediction of mutation effect, existing methods usually transform the problem to mini- de novo design, which infers a speciﬁc AA type from its microenvironment, or analogously its neighboring AA types.

Original: Current state-of-the-art  protein sequence-based methods rely heavily on multiple sequence alignment (MSA) (Riesselman et al., 2018; Frazer et al., 2021;
Revised: Current state-of-the-art sequencebased protein learning methods rely heavily on multiple sequence alignment (MSA; Riesselman et al. ; Frazer et al.

Original:  Rao et al., 2021) and protein language models (Elnaggar et al.,
Revised: ; Rao et al. and protein language models (Elnaggar et al.,

Original: ; Rives et al., 2021; Nijkamp et al., 2022; Brandes et al., 2022).
Revised: ; Rives et al., 2021; Nijkamp et al., 2022; Brandes et al., 2022).

Original: While MSA helps capture important evolutionary properties of the protein family, it nevertheless multiplies the requirements of computing resources.
Revised: While MSA helps capture important evolutionary properties of the protein family, it nevertheless multiplies the requirements of computing resources.

Original: The latter protein language models derived from natural language processing (NLP) encode sequence semantics and often need hundreds of GPU cards to train on billions of protein sequences.
Revised: The latter protein language models derived from natural language processing (NLP) encode sequence semantics and often need hundreds of GPU cards to train on hundreds of millions of proteins.

Original: Meanwhile, an auto-regressive inference process is usually required along the entire protein sequence to score a mutation on a single site, which further slows down the inference speed (Sato & Ishida, 2019;Meier et al., 2021;
Revised: Meanwhile, an autoregressive inference process is usually required along the entire protein sequence to score a mutation on a single site, which further slows down the inference speed (Sato & Ishida, 2019;

Original: Liu et al., 2022; Hsu et al., 2022; Notin et al., 2022).
Revised: Liu et al., 2022; Hsu et al., 2022; Notin et al., 2022).

Original: More importantly, when predicting the ﬁtness of the higher-order mutants, most of these models made a crude assumption that the multiple-site mutation effect is a linear summation of the effect of each individual mutation, which is incorrect in most cases (Lehner, 2011; Breen et al., 2012).
Revised: More importantly, when predicting the ﬁtness of the higher-order mutants, most of these models made a crude assumption that the mutations on different sites happen sequentially or individually, which is incorrect in most cases (Lehner, 2011; Breen et al., 2012).

Original: The ignored epistatic effects between different sites is potentially a key factor hindering the acquisition of favorable high-order mutants in directed evolution (Sarkisyan et al., 2016; Rollins et al., 2019).
Revised: The ignored epistatic effects between different sites are potentially a key factor hindering the acquisition of favorable high-order mutants in directed evolution (Sarkisyan et al., 2016; Rollins et al., 2019).

Original: Mutation of AA sites also occurs in nature, where an AA site might be mutated to any of the other19 AA types in a random manner.
Revised: Mutation of AA sites also occurs in nature, where an AA site might be mutated to any of the other AA types in a random manner.

Original: It is suggested by natural selection that only the mutation that exhibits the best ﬁtness and ﬁts the environment survives.
Revised: It is suggested by natural selection that only the mutants that exhibit the best ﬁtness and ﬁt the environment survive.

Revised: As a protein’s functionality is determined by its structure, we encode the folded protein by a protein graph with AAs being graph nodes to provide an elegant 3D spatial description of the protein.

Revised: The ﬁrst-level information, such as AA types, spatial coordinates of C ↵ , and C-N angles between neighboring AAs, are embedded in node features.

Original: Altering AA types of a protein in nature can  be viewed as adding corruptions to the node features of the protein graph, and denoising the graph makes a remedy to search for mutants with the best ﬁtness.
Revised: Altering AA types of a protein in nature can then be viewed as adding corruptions to the node features of the protein graph, and denoising the graph makes a remedy to search for mutants with the best ﬁtness.

Original: Wehereby model the protein mutation effect prediction as a denoising problem with equivariant graph neural networks (Satorras et al., 2021).
Revised: We model the protein mutation effect prediction as a denoising problem with equivariant graph neural networks (Satorras et al., 2021).

Original: Such a scheme predicts the ﬁtness of mutation effect on a protein with an arbitrary number of mutant sites andprovides efﬁcient recommendations to discover favorable mutants.
Revised: For a given protein, the recovered predictions can be leveraged to forecast the ﬁtness of deep mutational effects and discover favorable mutants.

Original: We treat each protein as a graph with AAs being graph nodes. The 3D spatial structure of the folded proteins is encoded with the graph geometry. The ﬁrst-level information, such asAA types, spatial coordinates of C ↵ , and C-N angles between neighboring AAs, are embedded in node features. We call the generated representation a protein graph , which provides an elegant structural description of the protein.
Revised: Compared to existing state-of-the-art deep learning methods for mutation effect prediction, such as

Original: Compared to existing zero-shot state-of-the-art methods for mutation effect prediction like ESM1 V (Meier et al., 2021) and ESM-IF1 (Hsu et al., 2022), the designed lightweight equivariant graph neural network (LGN) stands out in three perspectives.
Revised: ESM-1 V (Meier et al., 2021) and ESM-IF1 (Hsu et al., 2022), the designed lightweight equivariant graph neural network (LGN) stands out in three perspectives.

Original: First, our method improves generalization ability through the multi-task learning strategy and biological prior knowledge (the proposed model is fed with protein tertiary structure directly).
Revised: First, LGN improves generalization ability through the multi-task learning strategy and biological prior knowledge. The pre-trained model encodes the chemical and physical properties of a given

Original: The former trains data-driven black-box encoders to describe a given protein’s microenvironment, and the latter inserts domain knowledge towards practically meaningful representations. 
Revised: AA’s microenvironment with domain knowledge for practically meaningful representations. Secondly, LGN avoids the independent-mutation assumptions by generating the probabilities of all the amino acid residues at a time, which implements the joint distribution of all variations. In literature, the higher-order mutation effect is usually approached by summing up log-odd-ratio scores of the corresponding individual single-site mutants. The linear combination over separately assigned predictions is unsubstantiated, as the independent mutations neglect the epistatic effect.

Original: Secondly, the model is efﬁcient in both the training and inference phases.
Revised: Thirdly, LGN is efﬁcient in both the training and inference phases .

Original: The spatial graph inputs portray the topological properties of proteins, which circumvents data augmentation that is typically required by sequence or ﬁgure representations.
Revised: The spatial graph inputs portray the topological properties of proteins, which circumvents data augmentation that is typically required by sequence or grid representations.

Original: Equivariant message passing layers of the model, alternatively, provides a feature distillation unit with translationinvariance and rotation equivariance (Atz et al., 2021), which can precisely capture the micro-frame property in the protein graph geometry.
Revised: Equivariant message passing, alternatively, provides a feature distillation unit with translation and rotation equivariance and encodes AAs’ microenvironment deﬁned by the protein graph’s geometry.

Original:  Meanwhile, the downstream task allows outputting the probability of all the amino acid residues at a time, which is hundreds of times faster than classic auto-regressive methods. Thirdly, the typical approach to calculating the higher-order mutation effect sum up log-odd-ratio scores of the corresponding individual single-site mutants Meier et al. ; Hsu et al.
Revised:  Z ERO - SHOT L EARNING FOR P ROTEIN R ECOVERY

Original: However, the linear summation is unsubstantiated, as the epistatic effect is known to be present thatmutations of individual sites are not independent (Lehner, 2011; Breen et al., 2012). Instead, our scoring matrix does not need any of the independent-mutation assumptions, and the output can take any number of mutation sites at a time.
Revised: The excessive cost in laboratory results in scarce mutation scanning data, especially deep mutant results. It is thus favorable to pre-train a zero-shot protein prediction model that can be generalized directly to an unseen task without any further supervision to specialize the model.

Original: Z ERO - SHOT L EARNING FOR P ROTEIN R ECOVERY 2. G RAPH R EPRESENTATION OF P ROTEIN S TRUCTURE
Revised: 2.1 G RAPH R EPRESENTATION OF P ROTEIN S TRUCTURE

Original: For a given protein, we create a k-nearest neighbor (kNN) graph G =
Revised: For a given protein, we create a k-nearest neighbor (kNN) graph G =

Original: ( V , E ) to describe its 3D structure and molecular properties.
Revised: ( V , E ) to describe its 3D structure and molecular properties.

Original: Here each node v i 2 V represents an amino acid residue withSE(3)- invariant node attributes X 2 R 34  constituting biochemical properties and geometric properties of amino acids.
Revised: Here each node v i 2 V represents an amino acid residue with X 2 R 34 node attributes constituting biochemical properties and geometric properties of amino acids.

Original: The former includes 20 -dimensional attributes of one-hot encoded amino acid types ( X aa ), two scalars for each residue, i.e., solvent-accessible surface area (SASA) and the standardized crystallographic B-factor, and 5 normalized surface-aware node features.
Revised: The former includes 20 -dimensional attributes of one-hot encoded amino acid types ( X aa ), two scalars for each residue, i.e., solvent-accessible surface area (SASA) and the standardized crystallographic B-factor, and 5 normalized surface-aware node features.

Original: The geometric properties include the direction position ( X pos ) of each residue by 3D coordinates of its ↵ -carbon and the relative position of the amino acid in the protein chain ( X agl ) by the dihedral angles { sin , cos }  {  , } computed from the backbone atom positions.
Revised: The geometric properties include the direction position ( X pos ) of each residue by 3D coordinates of its ↵ -carbon and the relative position of the amino acid in the protein chain ( X agl ) by the dihedral angles { sin , cos }  {  , } computed from the backbone atom positions.

Original: For a speciﬁc v i of the i th amino acid in the protein sequence, the dihedral angles are measured from C ↵ i  1 , N i , C ↵ i , N i +1 .
Revised: For a speciﬁc v i of the i th amino acid in the protein sequence, the dihedral angles are measured from C ↵ i  1 , N i , C ↵ i , N i +1 .

Original: To build edge connections, we ﬁrst deﬁne a symmetric adjacency matrix A with the kNN-graph, i.e., each node is connected to up to k other nodes in the graph that has the smallest Euclidean distance over other nodes, and the distance is smaller than a certain cutoff, e.g., 30 ˚ A .
Revised: To build edge connections, we ﬁrst deﬁne a symmetric adjacency matrix A with the kNN-graph to capture the nodes’ microenvironment, i.e., each node is connected to up to k other nodes in the graph that has the smallest Euclidean distance over other nodes, and the distance is smaller than a certain cutoff ( 30 ˚ A ).

Original: Consequently, if v i and v j are connected to each other, we havee ij 2 E and A ij =
Revised: Consequently, if v i and v j are connected to each other, we have A ij =

Original: A ji 6 = 0 .
Revised: A ji 6 = 0 .

Original: The edge attributes E 2 R 93 are featured on the connected edges, including 15 inter-atomic distances, 12 local N-C positions, and66 -dimensional position encoding that records the relative position in the protein sequence.
Revised: The edge attributes E 2 R 93 feature the connected edges in E , including 15 inter-atomic distances, 12 local N-C positions, and the relative position in the protein sequence in 66 -dimensions.

Original: 2. P RE - TRAINING WITH P RIOR D OMAIN K NOWLEDGE FOR B ETTER P ROTEIN F ITNESS
Revised: 2.2 P RE - TRAINING WITH P RIOR D OMAIN K NOWLEDGE FOR B ETTER P ROTEIN F ITNESS

Original: The wild-type proteins suffer from random perturbations or mutations that not every AA site has the best AA type (Liu et al., 2022).
Revised: The wild-type proteins suffer from random perturbations or mutations that not every AA site has the best AA type (Liu et al., 2022).

Original: To this end, we pre-train our model with a multitask learning strategy, which removes the natural corruptions and predicts key protein properties to help encode the microenvironment of the stabilized proteins of interest.
Revised: To this end, we pre-train our model with a multitask learning strategy, which removes the natural corruptions and predicts key protein properties to help encode the microenvironment of the stabilized proteins of interest.

Original: The following section explains the perturbation and training strategies in accordance.

Original: AA Type Denoising For a node of amino type x aa , we reﬁne its type  ˜ x aa by adding a Bernoulli noise, i.e.,
Revised: AA Type Denoising We reﬁne x aa , the AA type a node, to ˜ x aa with a Bernoulli noise, i.e.,

Original: [Equation]
Revised: [Equation]

Original: where the conﬁdence level p is a tunable parameter that controls the proportion of residues that are ‘noise-free’.
Revised: where the conﬁdence level p is a tunable parameter that controls the proportion of residues that are ‘noise-free’.

Original: The probability for the residue to become a particular type depends on the distribution of the 20 types M ( n, ⇡ 1 , ⇡ 2 , ..., ⇡ n ) , which involves prior knowledge in molecular biology.
Revised: The probability for the residue to become a particular type depends on the distribution Under review as a conference paper at ICLRof the 20 types M ( n, ⇡ 1 , ⇡ 2 , ..., ⇡ n ) , which involves prior knowledge in molecular biology.

Original: This paper deﬁnes the distribution by the observed probability density of amino acid types in wild-type proteins 1 .
Revised: This paper deﬁnes the distribution by the observed probability density of amino acid types in wild-type proteins 1 .

Original: See Appendix E to better understand the inﬂuence of different conﬁdence levels.
Revised: See Appendix E to better understand the inﬂuence of different conﬁdence levels.

Original: Geometric Properties Denoising For the continuous-valued features, such as 3D coordinates and dihedral angles, an i.i.d Gaussian noise is introduced, learning to remove which corresponds to approximating the data-generating force ﬁeld of molecules (Zaidi et al., 2022).
Revised: Geometric Properties Denoising For the continuous-valued features, such as 3D coordinates and dihedral angles, an i.i.d Gaussian noise is introduced, learning to remove which corresponds to approximating the data-generating force ﬁeld of molecules (Zaidi et al., 2022).

Original: To be speciﬁc,
Revised: To be speciﬁc,

Original: [Equation]
Revised: [Equation]

Original:  where ✏ ⇠ N (0 , I [Equation] [Equation]
Revised: , where ✏ ⇠ N (0 , I 3 ) .

Original: The noise effect is determined by  , which is tunable to ﬁt the scale of the noiseless raw feature.
Revised: The noise effect is determined by  , which is tunable to ﬁt the scale of the noiseless raw feature.

Original: Bio-chemistry Properties Recovery Aside from  the perturbed residues type and geometric properties to denoise, other auxiliary tasks are introduced to better retrain the hidden microenvironment representation.
Revised: Bio-chemistry Properties Recovery Aside from denoising the perturbed residues type and ge- ometric properties, other auxiliary tasks are introduced to help establish an expressive hidden microenvironment representation.

Original: Speciﬁcally, SASA are known to strongly inﬂuence AA type preferences, and Bfactors are associated with the conformations and mobility of the neighboring AA.
Revised: Speciﬁcally, SASA is known to strongly inﬂuence AA type preferences, and B-factors are associated with the conformations and mobility of the neighboring AA.

Original: We thus introduces inductive biases to the model by predicting these two properties in the downstream tasks.
Revised: We thus introduce inductive biases to the model by predicting these two properties in the output.

Original: Label Smoothing with Amino Acid Substitution Matrices Protein sequence alignments provide important insights for understanding gene and protein functions.
Revised: Label Smoothing with Amino Acid Substitution Matrices Protein sequence alignments provide important insights for understanding gene and protein functions.

Original: The similarity measurement of an alignment of protein sequence reﬂects the favors of all possible exchanges of one amino acid with another.
Revised: The similarity measurement of an alignment of protein sequence reﬂects the favors of all possible exchanges of one amino acid with another.

Original: We employ BLOSUM (Henikoff & Henikoff, 1992), a substitution matrix , which accounts for the relative substitution frequencies and chemical similarity of amino acids, derived from the statistics for every conserved regions of protein families in BLOCKS database.
Revised: We employ BLOSUM (Henikoff & Henikoff, 1992), a substitution matrix, to account for the relative substitution frequencies and chemical similarity of AAs. The matrix is derived from the statistics for every conserved region of protein families in BLOCKS database.

Original: As AA sites are more likely to be mutated to the AA type within the block of high similarity scores in the BLOSUM table, we hereby modify our loss function  that a mutation to an AA type with a higher similarity score accumulates a smaller penalty than to the one with  lower similarity score.
Revised: As AA sites are more likely to be mutated to the AA type within the block of high similarity scores in the BLOSUM table, we hereby modify our loss function so that a mutation to an AA type with a higher similarity score accumulates a smaller penalty than to the one with a lower similarity score.

Original: 2. P ROTEIN S TRUCTURE R EPRESENTATION WITH E QUIVARIANT GNNS
Revised: 2.3 P ROTEIN S TRUCTURE R EPRESENTATION WITH E QUIVARIANT GNN

Revised: S

Original: As the initial proteins are structured in the 3-dimensional space,  it is vital for the model to predict the same binding complex no matter how the input proteins are positioned and oriented.
Revised: Proteins are structured in the 3-dimensional space, and it is vital for the model to predict the same binding complex no matter how the input proteins are positioned and oriented.

Original: Instead of practicing expensive data augmentation strategies, we follow Satorras et al.
Revised: Instead of practicing expensive data augmentation strategies, we follow Satorras et al.

Original: and construct
Revised: and construct

Original: SE(3)-equivariantand invariant neural layers for graph embedding.
Revised: SE(3)-equivariant neural layers for graph embedding.

Original: At the l th layer, an Equivariant Graph Convolution (EGC) inputs a set of n hidden node properties embedding H l =  h l 1 , . .
Revised: At the l th layer, an Equivariant Graph Convolution (EGC) inputs a set of n hidden node properties embedding H l =  h l 1 , . .

Original: , h ln as well as the node coordinate embeddings
Revised: , h ln as well as the node coordinate embeddings

Original: X l pos =
Revised: X l pos =

Original:  x l 1 , . . .
Revised:  x l 1 , . . .

Original: , x ln for a graph of n nodes.
Revised: , x ln for a graph of n nodes.

Original: The attributed edges are denoted as E = { . . .
Revised: The attributed edges are denoted as E = { . . .

Original: , e ij , . . . } .
Revised: , e ij , . . . } .

Original: The target of an EGC layer is to output a transformation on the node feature embedding H l +1 and coordinate embedding X l +1 pos .
Revised: The target of an EGC layer is to output a transformation on the node feature embedding H l +1 and coordinate embedding X l +1 pos .

Original: Concisely:
Revised: Concisely:

Original: H l +1 pos , X l +

Revised: H l +1 pos , X l +

Original: =
Revised: =

Original: EGC ⇥
Revised: EGC ⇥

Original: H l , X l pos [Equation] ⇤ .
Revised: H l , X l pos , E ⇤ .

Original: To achieve this, an EGC layer deﬁnes
Revised: To achieve this, an EGC layer deﬁnes

Original: [Equation]
Revised: [Equation]

Original: where  e ,  h are respectively the edge and node propagation operations, such as multi-layer perceptrons (MLPs).
Revised: where  e ,  h are respectively the edge and node propagation operations, such as multi-layer percep- trons (MLPs).

Original: The  x is an additional operation that projects the vector embedding m ij to a scalar value.
Revised: The  x is an additional operation that projects the vector embedding m ij to a scalar value.

Original: The EGC layer preserves equivariance to rotations and translations on the set of 3D node coordinates X pos , while simultaneously performing equivariance to permutations on the set of nodes V in the same fashion as GNNs.
Revised: The EGC layer preserves equivariance to rotations and translations on the set of 3D node coordinates X pos , while simultaneously performing invariance to permutations on the set of nodes V in the same fashion as GNNs.

Original: 2. M ODEL O VERVIEW
Revised: 2.4 M ODEL O VERVIEW

Original: Our model is depicted in Figure 1.
Revised: Our model is depicted in Figure 2.

Original: We take a set of protein graphs with node and edge attributes, and  each node’s 3D coordinates as the input of the pre-training model.
Revised: We take a set of protein graphs with attributed nodes and edges, as well as each node’s 3D coordinates, as the input to pre-train a zero-shot model.

Original: A stack of EGC layers is trained to extract rotationequivariant and translation invariant representations for each node on individual graphs.
Revised: A stack of EGC layers is trained to extract rotation and translation equivariant representations for each node on individual graphs.

Original: The hidden representation is then sent to fully-connected layers for downstream tasks, which contain multiple tasks such as AA type classiﬁcation, SASA and B-factor prediction, and 3D coordinates denoising.
Revised: The hidden representation is then sent to fully-connected layers to establish multiple outputs, such as AA type classiﬁcation, SASA and B-factor prediction, and 3D coordinates denoising.

Original: The total loss for the multitask learning task is given by
Revised: The total loss for the multitask learning task is given by

Original: [Equation]
Revised: [Equation]

Original: where  i , i = 1 , . .
Revised: where  i , i = 1 , . .

Original: . , 4 are tunable hyper-parameters to balance different losses on auxiliary regression tasks.
Revised: . , 4 are tunable hyper-parameters to balance different losses on auxiliary regression tasks.

Original: These losses are measured by mean squared error (MSE) loss.
Revised: These losses are measured by mean squared error (MSE) loss.

Original: For AA type classiﬁcation, we measure its loss L aa by cross-entropy with label smoothing technique (Szegedy et al., 2016).
Revised: For AA type classiﬁcation, we measure its loss L aa by cross-entropy with label smoothing technique (Szegedy et al., 2016).

Original: The classiﬁcation loss on an arbitrary node i reads
Revised: The classiﬁcation loss on an arbitrary node i reads

Original: [Equation]
Revised: [Equation]

Original: where p ( y aa |
Revised: where p ( y aa |

Original: X
Revised: X

Original: i , E i ) denotes the ground-truth distribution and q ✓ (ˆ y aa |
Revised: i , E i ) denotes the ground-truth distribution and q ✓ (ˆ y aa |

Original: X i , E i ) is the distribution of predicted labels following a softmax function.
Revised: X i , E i ) is the distribution of predicted labels following a softmax function.

Original: In order to improve the generalization and respect the prior biological knowledge, we modify the ground truth label distribution p ( y aa | X i , E i ) from the hard one-hot encoding to (1  " ) p 
Revised: In order to improve the generalization and respect the prior biological knowledge, we modify the ground truth label distribution p ( y aa | X i , E i ) from the hard one-hot encoding to (  " ) p ( y aa |

Original: ( y aa | X i , E i ) + " u ( y aa |
Revised: X i , E i ) + " u ( y aa |

Original: X i , E i ) when the predicted ˆ y aa
Revised: X i , E i ) when the predicted ˆ y aa

Original: =
Revised: =

Original: y aa and " u ( y aa |
Revised: y aa and " u ( y aa |

Original: X i , E i ) otherwise with some tolerance factor " .
Revised: X i , E i ) otherwise with some tolerance factor " .

Original: In particular, we deﬁne the distribution of u ( y | x i ) by the BLOSUM substitution matrix.
Revised: In particular, we deﬁne the distribution of u ( y | x i ) by the BLOSUM substitution matrix.

Original:  R ESULTS 3. E XPERIMENTAL S ETUP
Revised:  R ESULTS 3.1 E XPERIMENTAL S ETUP

Original: Our LGNis pre-trained on CATH v4.3.0 (Orengo et al., 1997) with artiﬁcial noise to predict AA type, 3D coordinates, dihedral angles, and chemical properties (SASA and B-factor).
Revised: We train LGN on CATH v4.3.0 (Orengo et al., 1997) with artiﬁcial noise to predict AA type, 3D coordinates, dihedral angles, and chemical properties (SASA and B-factor).

Original: The hidden embeddings of amino acids are learned by an equivariant GNN.
Revised: The hidden embeddings of amino acids are learned by SE(3)-equivariant graph convolutions.

Original: The evaluation is conducted with deep mutational scanning (DMS) datasets for zero-shot  ﬁtness of mutation prediction tasks.
Revised: The performance is validated by a zero-shot prediction task for the ﬁtness of mutation prediction with deep mutational scanning (DMS; Fowler & Fields (2014)) datasets.

Original: The model performance is compared against popular state-of-the-art language models and structureenhanced models.
Revised: The model performance is compared against popular state-of-the-art language models and structure-enhanced models.

Original: Baseline Models We compare with a diverse of state-of-the-art models on the ﬁtness of mutation effects prediction.
Revised: Baseline Models We compare with a diverse of state-of-the-art models on the ﬁtness of mutation effects prediction.

Revised: In particular, D EEP S EQUENCE (Riesselman et al., 2018) trains VAE on proteinspeciﬁc MSAs to capture higher-order interactions from the distribution of an AA sequence.

Original: In particualr, MSA T RANSFORMER Rao et al.
Revised: MSA T RANSFORMER Rao et al.

Original: (2021) is a language model with aligned protein sequences of interest; ESM-1 V (Meier et al., 2021) make zero-shot mutation predictions with masked language modeling; and ESM-IF1 (Hsu et al., 2022) predicts protein sequence with GVP, a graph representation learning methods for vector and scalar features of protein graphs.
Revised: (2021) is a language model with aligned protein sequences of interest; ESM-1 V (Meier et al., 2021) make zero-shot mutation predictions with masked language modeling; and ESM-IF1 (Hsu et al., 2022) predicts protein sequence with GVP (Jing et al., 2020), a graph representation learning methods for vector and scalar features of protein graphs.

Original: The model is pre-trained on a mixture of CATH and AlphaFold2 datasets, and the sequence embedding is enhanced with transformers (Vaswani et al., 2017).
Revised: Furthermore, both T RANCEPTION (Notin et al., 2022) and P RO G EN 2 (Nijkamp et al., 2022) leverages autoregressive language models to retrieve AA sequence without family-speciﬁc MSAs.

Original: Lightweight Equivariant Graph Neural Networks (LGN)
Revised: Lightweight Equivariant Graph Neural Networks (LGN)

Original: To train our LGN framework, we ﬁrst generate protein graphs for the sequences in CATH .
Revised: To train our LGN framework, we ﬁrst generate protein graphs for the sequences in CATH .

Original: See Appendix A for a detailed introduction on the generation, and Appendix C for summaries in the generated dataset.
Revised: See Appendix A for a detailed introduction to the generation, and Appendix C for a summary of the generated dataset.

Original: For the total number of 31 , 848 protein graphs of 150 nodes on average, we randomly pick 500 graphs for validation and leave the remaining for model ﬁtting.
Revised: For the total number of 31 , 848 protein graphs of 150 nodes on average, we randomly pick 500 graphs for validation and leave the remaining for model ﬁtting.

Original: During the learning phase, we assign random perturbations to AA types and other features we mentioned earlier in Section 2.
Revised: During the learning phase, we assign random perturbations to AA types and other features we mentioned earlier in Section 2.

Original: During the validation step, the noises are ﬁxed for stable and comparable measurements.
Revised: The noises are ﬁxed to guarantee stable and comparable measurements at the validation step.

Original: The speciﬁc inﬂuence on different choices on the hyper-parameters (e.g., the noise level) will be discussed later in this section and Appendix DE.
Revised: The speciﬁc inﬂuence of different choices on the hyper-parameters (e.g., the noise level) will be discussed later in this section and Appendix D-E.

Original: The main architecture constitutes a stack of 6 EGC layers following 1 fully-connected layer to make predictions on the different learning tasks.
Revised: The main architecture constitutes a stack of 6 EGC layers following 1 fully-connected layer to make predictions on the different learning tasks.

Original: On each node, the output is a vector representation consisting of 20 probabilities of the masked amino acid, 1 predicted SASA, 1 B-factors,  and  dihedral values (when applicable).
Revised: On each node, the output is a vector representation consisting of 20 probabilities of the masked amino acid, 1 predicted SASA, 1 B-factors, [Table] and 4 dihedral values (when applicable).

Original: The 3D-coordinates are derived directly from EGC outputs.
Revised: The 3D coordinates are derived directly from EGC outputs.

Original: The loss function by Equation 4 guides the backward propagation with A DAM (Kingma & Ba, 2015) optimizer.
Revised: The loss function by Equation 4 guides the backward propagation with A DAM (Kingma & Ba, 2015) optimizer.

Original: The model is trained with 300 epochs with the initial rate set to 0 .
Revised: The model is trained with 300 epochs with the initial rate set to 0 .

Original: 001 and weight decay to 0 .
Revised: 001 and weight decay to 0 .

Original: The learning rate isfurther dampened to 0 .
Revised: The learning rate is dampened to 0 .

Original: 0001 after 150 epochs.
Revised: 0001 after 150 epochs.

Original: Evaluation All the models are evaluated with deep mutational scanning (DMS) assays that assess a diverse set of 15 proteins, where 9 of them only contains single-site mutation scores, and 6 of them have both single-site and higher-order mutant records (see Appendix B).
Revised: Evaluation All the models are evaluated with deep mutational scanning (DMS) assays that assess a diverse set of 15 proteins, where 9 of them only contains single-site mutation scores [Table] and 6 of them have both single-site and higher-order mutational records (see Appendix B).

Original: The protein structures are folded from the provided sequence information with A LPHA F OLD  (Jumper et al., 2021), following the exact same pre-processing steps as in CATH for generating protein graphs.
Revised: The protein structures are folded from the provided sequence information with A LPHA F OLD 2 (Jumper et al., 2021), fol- lowing the exact same pre-processing steps as in CATH for generating protein graphs.

Original: The only difference is that we do not append artiﬁcial noises onto the test proteins, as we assume they are already noisy.
Revised: The only difference is that we do not append artiﬁcial noises onto the test proteins, as we assume they are already noisy.

Original: We then send the unmutated test proteins to the pre-trained LGN model and use the log-odd-ratio in Equation 5 of the predicted probabilities of AA types for suggesting the  deep mutant ranks.
Revised: We then send the unmutated test proteins to the pre-trained LGN model and use the log-odd-ratio in Equation 5 of the predicted probabilities of AA types for suggesting the rank of deep mutations.

Original: Theﬁnal performance on deep mutant prediction  is evaluated on Spearman’s correlation coefﬁcient between the predicted and experimental scores on all the mutation combinations in the deep scanning dataset.
Revised: The prediction performance is evaluated on Spearman’s correlation coefﬁcient between the computational and experimental scores on all the mutation combinations.

Original: 3. F ITNESS OF D EEP M UTANTS P REDICTION
Revised: 3.2 F ITNESS OF D EEP M UTANTS P REDICTION

Original: The ﬁrst experiment evaluates the ﬁtness of proteins’ mutation effects prediction with zero-shot models, i.e., the ﬁtness scores are inferred directly from a pre-trained model without ﬁne-tuning on a task-speciﬁc model.
Revised: The ﬁrst experiment evaluates the ﬁtness of proteins’ mutation effects prediction, where the ﬁtness scores are inferred directly from a pre-trained model without supervision on a task-speciﬁc model.

Original: We visualize the overall performance comparison on protein-wise Spearman correlation coefﬁcients in Figure 3.
Revised: We visualize the overall performance comparison on protein-wise Spearman’s correlation coefﬁcients in Figure 3.

Original: Our LGN outperforms baseline methodsin deep mutant tasks and achieves at least comparable results in single-site mutant tests.
Revised: The deep mutation scores are reported in Table 1, where LGN outperforms baseline methods and achieves at least comparable results in single-site mutant tests.

Original: Overall, our model achieves 0 .
Revised: Overall, our model achieves 0 .

Original: 5037 weighted average correlations on deep mutant effect prediction by predicting AA types and SASA, and 0 .
Revised: 5071 weighted average correlations on deep mutant effect predictions. While

Original: 4819 on AA types, B-factor and SASA. In contrast, MSA T RANSFORMER only achieves 0 . 3756 Spearman’s correlation. The scores for ESM-IF1 and ESM-1 V are 0 .and 0 .
Revised: D EEP S EQUENCE achieves superior performance over the majority rest, it should be noticed that the model has to be trained on every new protein, and it cannot be generated for other proteins.

Original: 3902 , respectively. For the single-site mutations, we also achieved at-least comparable performance to the baseline methods at a signiﬁcantly less computational cost.
Revised: Also, the training speed and performance of the learned model rely heavily on the quality of the available MSA information, which can very a lot on different proteins.

Original: 3. P ROTEIN R ECOVERY
Revised: 3.3 P ROTEIN R ECOVERY

Original: In this experiment, we investigate the auxiliary learning tasks of the pre-trained model, including the learning performance in predicting AA types, SASA, and B-factor.
Revised: This experiment investigates the auxiliary learning tasks of the pre-trained model, including the learning performance in predicting AA types, SASA, and B-factor.

Original: In speciﬁc, we visualize the confusion matrix of the predicted AA types with respect to the ground-truth AA types to see evaluate the model’s capability to recover from noisy sequences to the original sequence, i.e., if the large values are accumulated to the diagonal of the confusion matrix.
Revised: In speciﬁc, we visualize the confusion matrix of the predicted AA types with respect to the ground-truth AA types to see evaluate the model’s capability to recover from noisy sequences to the original sequence, i.e., if the large values are accumulated to the diagonal of the confusion matrix.

Original: For the SASA and Bfactor predictions, we examine the R 2 of the predicted and the ground-truth values on CATH .
Revised: For the SASA and B-factor predictions, we examine the R 2 of the predicted and the ground-truth values on CATH .

Original: The results are visualized in Figure 4 with denoised AA type, as well as the predicted SASA and Bfactor as the output tasks.
Revised: The results are visualized in Figure 4 with denoised AA type, as well as the predicted SASA and B-factor as the output tasks.

Original: The AA type prediction achieves high accuracy with the majority of predictions accumulated on the diagonal line.
Revised: The AA type prediction achieves high accuracy with the majority of predictions accumulated on the diagonal line.

Original: For the two regression tasks, we ﬁt the true value and the predicted value with linear regression.
Revised: For the two regression tasks, we ﬁt the true value and the predicted value with linear regression.

Original: The estimated coefﬁcients are 1 . 008 and 0 .989 for
Revised: The estimated coefﬁcients are 1 . 008 and 0 .

Original:  B-factor and SASA, respectively.
Revised: 989 for B-factor and SASA, respectively.

Original: The p -value for both coefﬁcients is  0 .
Revised: The p -value for both coefﬁcients is < 0 .

Original: In addition, Pearson’s correlation coefﬁcients for the two variants are respectively 0 . 884 and 0 .
Revised: In addition, Pearson’s correlation coefﬁcients for the two variants are respectively 0 . 884 and 0 .

Original: 3. S ELECTION C RITERIA ON THE A DDITIONAL H YPER - PARAMETERS
Revised: 3.4 S ELECTION C RITERIA ON THE A DDITIONAL H YPER - PARAMETERS

Original: As LGN introduces two additional hyper-parameters in training the model, this section examines the inﬂuence of selecting different p s and  s. Figure 5 visualizes a selection of model performance on the Spearman’s correlation with variant p s and  s with wild-type noise distribution.
Revised: As LGN introduces two additional hyper-parameters in training the model, this section examines the inﬂuence of selecting different p s and  s. Figure 5 visualizes a selection of model performance on the Spearman’s correlation with variant p s and  s with wild-type noise distribution.

Original: While the choice of p can be determined by prior knowledge regarding the quality of wild-type proteins, here we treat p as a data-driven hyper-parameter to be optimized during the model training.
Revised: While the choice of p can be determined by prior knowledge regarding the quality of wild-type proteins, here we treat p as a data-driven hyper-parameter to be optimized during the model training.

Original: We exclude extremely small p s to avoid drastic perturbation rates and search for the optimal p{ 0 . 3 , 0 . 4 , . . .
Revised: We exclude extremely small p s to avoid drastic perturbation rates and search for the optimal p{ 0 . 3 , 0 . 4 , . . .

Original: The different choices on p s are validated with different learning tasks on the left side of Figure 5 for higher-order mutants.
Revised: The different choices on p s are validated with different learning tasks on the left side of Figure 5 for higher-order mutants.

Original: In general, a moderate p between 0 . 3 and 0 .
Revised: In general, a moderate p between 0 . 3 and 0 .

Original: 6 best suits the majority selection of learning modules and noise distribution.
Revised: 6 best suits the majority selection of learning modules and noise distribution.

Original: Based on the overall performance, we suggest p = 0 .
Revised: Based on the overall performance, we suggest p = 0 .

Original: 6 as the default value of the conﬁdence level (See Appendix E for more results on different types of perturbation noise and proteins).
Revised: 6 as the default value of the conﬁdence level (See Appendix E for more results on different types of perturbation noise and proteins).

Original: We also investigate a wide range of the choices of  s. For simplicity, we let  

Revised: We also investigate a wide range of the choices of  s. For simplicity, we let

Original: =
Revised: =

Original:  2 =  3{ 0 . 05 , 0 . 1 , 0 . 2 , 0 .
Revised:  2 =  3{ 0 . 05 , 0 . 1 , 0 . 2 , 0 .

Original: 5 , 0 . 8 , 5 , 10 } and ﬁx  4 = 0 .
Revised: 5 , 0 . 8 , 5 , 10 } and ﬁx  4 = 0 .

Original: All the results are conducted under the recommended p = 0 .
Revised: All the results are conducted under the recom- mended p = 0 .

Original: 6 with wild-type noise.
Revised: 6 with wild-type noise.

Original: We report the average performance on deep mutants in the  right plot of Figure 5, which demonstrates a relatively ﬂat and steady trend with a mild peak at  = 0 .
Revised: We report the average performance on deep mutants in the [Table] right plot of Figure 5, which demonstrates a relatively ﬂat and steady trend with a mild peak at  = 0 .

Original: We supplement additional experimental results  in Table 7 of Appendix E with different model setups.
Revised: Additional results are provided in Table 8 of Appendix E with various model setups [Table]

Original: 3. I NFERENCE S PEED
Revised: 3.5 I NFERENCE S PEED

Original: Our LGN consumes signiﬁcantly fewer computational resources in training and inference.
Revised: LGN consumes signiﬁcantly fewer computational resources in training and inference.

Original: This section reports a direct comparison of the model scale, inference time, and prediction performance. The results are visualized in Figure 6. In Table 
Revised: We compare the model scale, inference time, and prediction performance in Figure 6 and Table 2.

Original: [Table] we compare the model size and the required resources with the baseline methods: MSA
Revised: The model size and the required resources with the baseline methods are provided by the authors.

Original: TRANSFORMER (Rao et al., 2021), ESM-1 V (Meier et al., 2021), and ESM-IF1 (Hsu et al., 2022). To be speciﬁc, the attributes for training the model are provided by the authors, and the inference speed and memory are tested by us.
Revised: As the majority of models are pre-trained, we record the inference speed on a single 3090 GPU. While the time cost is signiﬁcantly lower than experimental methods, we measure it to indicate the cost of forward propagation in one iteration, which can be viewed as the indirect empirical evidence of the training cost.

Original: As each protein requires independent inference progress [Table] we hereby take GFP as an example protein sample.
Revised: As each protein requires independent inference progress, we hereby take GFP as an example protein sample.

Original: The protein constitutes 236 amino acid residues, and it has over 50 , 000 mutant records. (see Table 2 for more details).
Revised: The protein constitutes 236 amino acid residues, and it has over 50 ,mutant records (see Table 3 in Appendix B for more details).

Original: Note that: 
Revised: Note that: 1).

Original: the 2 , 687 input token length only refers to the maximum protein length we used during training.
Revised: The 2 , 687 input token length only refers to the maximum protein length we used during training.

Original: In fact, the model itself can process large protein graphs containing over tens of thousands of amino acids. 
Revised: In fact, the model itself can process large protein graphs containing over tens of thousands of amino acids. 2).

Original: the training speed and required resources for MSA TRANSFORMER are retrieved from Meier et al.
Revised: The training speed and required resources for MSA TRANSFORMER are retrieved from Meier et al.

Original: The original work by Rao et al.
Revised: The original work by Rao et al.

Original: (2021) only reports that they used 32 ⇥ V100 GPUs for training, without revealing the training time.[Table]
Revised: (2021) only reports that they used⇥ V100 GPUs for training, without revealing the training time.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Protein Sequence and Structure Representation
Revised: Protein Sequence and Structure Representation

Original: Due to the enormous experimental cost of measuring protein structures, the number of known protein sequences is thousands of times larger than protein structures (Eswar et al., 2008; Hsu et al., 2022).
Revised: Due to the enormous experimental cost of mea- suring protein structures, the number of known protein sequences is thousands of times larger than Under review as a conference paper at ICLRprotein structures (Eswar et al., 2008; Hsu et al., 2022).

Original: Meanwhile, the protein sequences representation is highly similar to human language, which naturally promotes the fast development of natural language processing (NLP), especially transformer-based methods for encoding protein sequences (Coin et al., 2003; Meier et al., 2021; Ofer et al., 2021).
Revised: Meanwhile, the protein sequences representation is highly similar to human language, which naturally promotes the fast development of natural language processing (NLP), especially transformer-based methods for encoding protein sequences (Coin et al., 2003; Meier et al., 2021; Ofer et al., 2021; Castro et al., 2022).

Original: However, the geometry of proteins also suggests higher-level structures and topological relationships that are vital to protein functionality.
Revised: However, the geometry of proteins also suggests higher-level structures and topological relationships that are vital to protein functionality.

Original: Structure prediction of proteins always attracts great attention in the ﬁeld (Chi & Liberles, 2016;
Revised: Structure prediction of proteins always attracts great attention in the ﬁeld (Chi & Liberles, 2016;

Original: Jumper et al., 2021; Baek et al., 2021; Varadi et al., 2022).
Revised: Jumper et al., 2021; Baek et al., 2021; Varadi et al., 2022).

Original: The breakthrough progress in protein folding also enriches structured proteins.
Revised: The breakthrough progress in protein folding also enriches structured proteins.

Original: For instance, Hsu et al.
Revised: For instance, Hsu et al.

Original: and
Revised: and

Original: Ma et al.
Revised: Ma et al.

Original: (2022) mixed experimentally-tested and A LPHA F OLD -predicted for model training, which greatly eases the data shortage problem and achieves signiﬁcant performance gain.
Revised: (2022) mixed experimentally-tested and A LPHA F OLD -predicted for model training, which greatly eases the data shortage problem and achieves signiﬁcant performance gain.

Original: Structural encoding for Protein Graphs According to the laws of physics, the atomic dynamics do not changed no matter how a protein is translated or rotated from one place to another (Han et al., 2022).
Revised: Structural encoding for Protein Graphs According to the laws of physics, the atomic dynamics do not change no matter how a protein is translated or rotated from one place to another (Han et al., 2022).

Original: Therefore, the inductive bias of symmetry should be incorporated into the design of protein structure-based models.
Revised: Therefore, the inductive bias of symmetry should be incorporated into the design of protein structure-based models.

Original: To this end, research work has been proposed to respect the spatial relationship of amino acids (Torng & Altman, 2017; Sato & Ishida, 2019).
Revised: To this end, research work has been proposed to respect the spatial relationship of amino acids (Torng & Altman, 2017; Sato & Ishida, 2019).

Original: Such CNN-based methods aggregate the local structure of each residue and integrate estimated local qualities into the whole protein properties.
Revised: Such CNN-based methods aggregate the local structure of each residue and integrate estimated local qualities into the whole protein properties.

Original: However, these methods neglect geometric equivariance, which can usually be captured by equivariant graph neural networks (Ganea et al., 2021; St¨ark et al., 2022).
Revised: However, these methods neglect geometric equivariance, which can usually be captured by equivariant graph neural networks (Ganea et al., 2021; St¨ark et al., 2022).

Original: Protein Representation As existing protein language models require high computational costs and are difﬁcult to train, ﬁnding an effective feature representation of protein data is important for downstream tasks (Thompson et al., 2012).
Revised: Protein Representation As existing protein language models require high computational costs and are difﬁcult to train, ﬁnding an effective feature representation of protein data is important for downstream tasks (Thompson et al., 2012).

Original: Contrastive learning and self-prediction Elnaggar et al. (2021), Zhang et al. andHsu et al. (2022) used self-supervised pre-training methodsprovide ways to extract good representation for reducing computational resources.
Revised: Contrastive learning and self-prediction (Elnaggar et al., 2021; Zhang et al., 2022; Hsu et al., 2022) used self-supervised pre-training methods to extract good representation for reducing computational resources.

Original: Despite only applying classical representations learning methods on protein, some researchers designed sophisticated methods.
Revised: Despite only applying classical representation learning methods on protein, some researchers designed sophisticated encoders for expressive protein representation.

Original:  Li et al.
Revised: For instance, Li et al.

Original: (2022) proposed ~
Revised: (2022) proposed ~

Original: W -GNN variants  efﬁciently interact  scalar-vector features.
Revised: W -GNN variants that efﬁciently interact with scalar-vector features.

Original: Besides, Somnath et al.
Revised: Somnath et al.

Original: (2021) introduceda multi-scale model HOLOPROTconnecting surface to structure and sequence.
Revised: (2021) introduced HOLOPROT to connect different modalities of proteins, including surface, structure, and sequence representation.

Original: Mutation Effect Prediction Multiple sequence alignment (MSA) is an essential ingredient for many of the existing state-of-the-art methods to predict the effect of single amino acid substitutions such as D EEP S EQUENCE (Riesselman et al., 2018), A LPHA F OLD  (Jumper et al., 2021) and MSA 
Revised: Mutation Effect Prediction Multiple sequence alignment (MSA) is an essential ingredient for many of the existing state-of-the-art methods to predict the effect of single amino acid substitutions such as D EEP S EQUENCE (Riesselman et al., 2018), A LPHA F OLD 2 (Jumper et al., 2021), MSA T

Original: T RANSFORMER (Rao et al., 2021).
Revised: RANSFORMER (Rao et al., 2021), and LM-GVP (Wang et al., 2022).

Original: The MSA for a protein sequence or domain captures meaningful information on  evolutionary information of the protein within its family at the cost of bringing severe limitations–not all proteins are alignable, such as CDRs of antibody variable domains (Shin et al., 2021), and not all the alignments are deep enough to train models sufﬁciently large to learn the complex interactions between residues.
Revised: The MSA for a protein sequence or domain captures meaningful information on the evolutionary information of the protein within its family at the cost of bringing severe limitations–not all proteins are alignable, such as CDRs of antibody variable domains (Shin et al., 2021), and not all the alignments are deep enough to train models sufﬁciently large to learn the complex interactions between residues.

Original: To deal with this issue, ESM-1 V (Meier et al., 2021) trains a zero-shot model on a large set of unaligned sequences to secure a scalable and bias-free training procedure, and T RANCEPTION (Notin et al., 2022) leverages autoregressive pre- dictions and retrieval of homologous sequences at inference.
Revised: To deal with this issue, ESM-1 V (Meier et al., 2021) trains a zero-shot model on a large set of unaligned sequences to secure a scalable and bias-free training procedure, and T RANCEPTION (Notin et al., 2022) leverages autoregressive predictions and retrieval of homologous sequences at inference.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Designing directed evolution on proteins especially the deep mutants for functional ﬁtness is of enormous engineering and pharmaceutical importance.
Revised: Designing directed evolution on proteins, especially with deep mutants for functional ﬁtness, is of enormous engineering and pharmaceutical importance.

Original: However, existing experimental methods aretime and economically costing.
Revised: However, existing experimental methods are economically costly, and in silico methods require signiﬁcant computational resources.

Original: In this paper, we proposed a lightweight graph neural network model(LGN) for mutant effect prediction on arbitrary numbers of protein mutations by transferring the problem to denoising  protein graphs.
Revised: This paper proposed a lightweight zero-shot model for mutant effect prediction on arbitrary numbers of AAs by transferring the problem to denoising a protein graph.

Original: We assume the proteins in nature are noisy protein graphs and we train our model  to recover amino acid types and other important properties (i.e., b-factor, SASA,  the spatial position of C ↵ , and dihedral angles of neighboring N-C atoms) from observed noisy proteins.
Revised: Our model is trained to recover AA types and other important properties (e.g., B-factor, SASA, and the spatial position of C ↵ ) from observed noisy proteins.

Original: Translation invariant and rotation equivariant graph convolution layers in our model extract  the invariant geometry features of micro frames within and between amino acids and thus grasp rich information for efﬁciently learning protein function.
Revised: We employ translation and rotation equivariant neural message passing layers to extract geometric-aware representation for the microenvironment of central AAs and thus grasp rich information for efﬁciently learning protein function.

Original: The model achieves state-of-theart performance on PDB datasets  with signiﬁcantly fewer computational resources than existing  models.Especially in the deep mutant test, our LGN outperforms other zero-shot models by over 20%.
Revised: The model achieves state-of-the-art performance on PDB datasets in deep mutant tests with signiﬁcantly fewer computational resources than existing SOTA models.