Original: Exploring evolution-aware & -free protein language models as protein function predictors
Revised: Exploring evolution-aware & -free protein language models as protein function predictors

Original: Abstract
Revised: Abstract

Original: Large-scale Protein Language Models (PLMs) have improved performance in protein prediction tasks, ranging from 3D structure prediction to various function predictions.
Revised: Large-scale Protein Language Models (PLMs) have improved performance in protein prediction tasks, ranging from 3D structure prediction to various function predictions.

Original: In particular, AlphaFold [23], a ground-breaking AI system, could potentially reshape structural biology.
Revised: In particular, AlphaFold [23], a ground-breaking AI system, could potentially reshape structural biology.

Original: However, the utility of the PLM module in
Revised: However, the utility of the PLM module in

Original: AlphaFold, Evoformer, has not been explored beyond structure prediction.
Revised: AlphaFold, Evoformer, has not been explored beyond structure prediction.

Original: In this paper, we investigate the representation ability of three popular PLMs: ESM-1b (single sequence) [35], MSA-Transformer (multiple sequence alignment) [30] and
Revised: In this paper, we investigate the representation ability of three popular PLMs: ESM-1b (single sequence) [35], MSA-Transformer (multiple sequence alignment) [30] and

Original: Evoformer (structural), with a special focus on Evoformer.
Revised: Evoformer (structural), with a special focus on Evoformer.

Original: Specifically, we aim to answer the following key questions: (i) Does the Evoformer trained as part of
Revised: Speciﬁcally, we aim to answer the following key questions: (i) Does the Evoformer trained as part of

Original: AlphaFold produce representations amenable to predicting protein function?
Revised: AlphaFold produce representations amenable to predicting protein function?

Original: (ii)
Revised: (ii)

Original: If yes, can Evoformer replace ESM-1b and MSA-Transformer?
Revised: If yes, can Evoformer replace ESM-1b and MSA-Transformer?

Original: (iii) How much do these PLMs rely on evolution-related protein data?
Revised: (iii) How much do these PLMs rely on evolution-related protein data?

Original: In this regard, are they complementary to each other?
Revised: In this regard, are they complementary to each other?

Original: We compare these models by empirical study along with new insights and conclusions.
Revised: We compare these models by empirical study along with new insights and conclusions.

Original: All code and datasets for reproducibility are available at https://github.com/elttaes/Revisiting-PLMs .
Revised: All code and datasets for reproducibility are available at https://github.com/elttaes/Revisiting-PLMs .

Original:  Introduction
Revised:  Introduction

Original: Proteins perform the majority of biological activities.
Revised: Proteins perform the majority of biological activities.

Original: It is, therefore, crucial to decipher the mechanisms underlying their structural and functional properties.
Revised: It is, therefore, crucial to decipher the mechanisms underlying their structural and functional properties.

Original: The canonical sequence-structure-function relationship enables the success of sequence-based machine learning methods that infer protein structure and function from amino acid (AA) sequence.
Revised: The canonical sequence-structure-function relationship enables the success of sequence-based machine learning methods that infer protein structure and function from amino acid (AA) sequence.

Original: Large-scale protein language models (PLMs) with self-supervised pretraining on tens of millions to billions of proteins (PLMs) [28, 35, 13, 5] are the current state-of-the-art in predicting function and fitness from sequences.
Revised: Large-scale protein language models (PLMs) with self-supervised pretraining on tens of millions to billions of proteins (PLMs) [28, 35, 13, 5] are the current state-of-the-art in predicting function and ﬁtness from sequences.

Original: Meanwhile, AlphaFold [23], trained on experimental 3D protein structures from the Protein Data
Revised: Meanwhile, AlphaFold [23], trained on experimental 3D protein structures from the Protein Data

Original: Bank (PDB) [39] can approach the resolution of experimental structures for most protein sequences.
Revised: Bank (PDB) [39] can approach the resolution of experimental structures for most protein sequences.

Original: Its multiple sequence alignment representation module, Evoformer, combines new deep learning machinery, a PLM residue reconstruction task, and structural supervision in the form of a distogram.
Revised: Its multiple sequence alignment representation module, Evoformer, combines new deep learning machinery, a PLM residue reconstruction task, and structural supervision in the form of a distogram.

Original: Like MSA-Transformer [30], Evoformer takes a family of evolutionarily-related and aligned protein sequences as input, in contrast to PLMs such as ESM-1b [35] and TAPE [28], which only take individual protein sequences.
Revised: Like MSA-Transformer [30], Evoformer takes a family of evolutionarily-related and aligned protein sequences as input, in contrast to PLMs such as ESM-1b [35] and TAPE [28], which only take individual protein sequences.

Original: Thus, for short, we refer to the former two models as evolution-aware
Revised: Thus, for short, we refer to the former two models as evolution-aware

Original: PLMs and the latter two as evolution-free PLMs.
Revised: PLMs and the latter two as evolution-free PLMs.

Original: Despite the remarkable success of AlphaFold in predicting structure from sequence, it is unknown whether its Evoformer module can be applied to other problems, in particular predicting protein function and fitness.
Revised: Despite the remarkable success of AlphaFold in predicting structure from sequence, it is unknown whether its Evoformer module can be applied to other problems, in particular predicting protein function and ﬁtness.

Original: Deciphering AlphaFold rather than treating it as a black box is beneficial to both
Revised: Deciphering AlphaFold rather than treating it as a black box is beneﬁcial to both

Original: AI and biology communities.
Revised: AI and biology communities.

Original: Therefore, we attempt to answer the following key questions.
Revised: Therefore, we attempt to answer the following key questions.

Original: Q(i): Does the Evoformer trained during AlphaFold training learn general-purpose protein representations that can be used for various function prediction tasks?
Revised: Q(i): Does the Evoformer trained during AlphaFold training learn general-purpose protein representations that can be used for various function prediction tasks?

Original: Unlike ESM-1b and MSATransformer, Evoformer is trained with supervision from 3D structures.
Revised: Unlike ESM-1b and MSATransformer, Evoformer is trained with supervision from 3D structures.

Original: In addition, the second part of AlphaFold, the Structure Module, built on top of the 48 Evoformer blocks, is much more complex and deeper than the traditional (linear) classification head used in ESM-1b and MSA-Transformer.
Revised: In addition, the second part of AlphaFold, the Structure Module, built on top of the 48 Evoformer blocks, is much more complex and deeper than the traditional (linear) classiﬁcation head used in ESM-1b and MSA-Transformer.

Original: These differences make the function representation ability of Evoformer an open question.
Revised: These differences make the function representation ability of Evoformer an open question.

Original: Q(ii):
Revised: Q(ii):

Original: If Evoformer’s representation is general, does it outperform ESM-1b and MSATransformer on downstream tasks?
Revised: If Evoformer’s representation is general, does it outperform ESM-1b and MSATransformer on downstream tasks?

Original: While these three models are trained with different parameter sizes and datasets (see Table 1), we believe the comparison results are still valuable because they are currently the most advanced PLMs.
Revised: While these three models are trained with different parameter sizes and datasets (see Table 1), we believe the comparison results are still valuable because they are currently the most advanced PLMs.

Original: Training these large models from scratch is out of the reach of most academic research groups due to the computation and cost involved.
Revised: Training these large models from scratch is out of the reach of most academic research groups due to the computation and cost involved.

Original: In addition, we also investigate the utility of MSA-Transformer on function prediction tasks.
Revised: In addition, we also investigate the utility of MSA-Transformer on function prediction tasks.

Original: Q(iii): How much does the performance of evolution-aware PLMs rely on the input MSAs?
Revised: Q(iii): How much does the performance of evolution-aware PLMs rely on the input MSAs?

Original: Can evolution-free PLMs assist evolution-aware PLMs in terms of MSA construction?
Revised: Can evolution-free PLMs assist evolution-aware PLMs in terms of MSA construction?

Original: Both
Revised: Both

Original: Evoformer and MSA-Transformer take sets of aligned sequences (MSAs) as input.
Revised: Evoformer and MSA-Transformer take sets of aligned sequences (MSAs) as input.

Original: We investigate the effect of MSA quality and depth on function prediction.
Revised: We investigate the effect of MSA quality and depth on function prediction.

Original: On the other side, ESM-1b can simply formulate MSA construction as the remote homology detection task.
Revised: On the other side, ESM-1b can simply formulate MSA construction as the remote homology detection task.

Original: Then, it is interesting to know whether ESM1b-constructed MSAs can be used as inputs to Evoformer and MSA-Transformer.
Revised: Then, it is interesting to know whether ESM-1b-constructed MSAs can be used as inputs to Evoformer and MSA-Transformer.

Original: We address the above questions through comprehensive empirical studies on a variety of structure and function prediction tasks.
Revised: We address the above questions through comprehensive empirical studies on a variety of structure and function prediction tasks.

Original: Note although AlphaFold (i.e. Evoformer + Structure Module) can accurately predict protein 3D structures, we investigate the ability of Evoformer to perform other structure prediction tasks, such as secondary structure prediction and contact map prediction.
Revised: Note although AlphaFold (i.e. Evoformer + Structure Module) can accurately predict protein 3D structures, we investigate the ability of Evoformer to perform other structure prediction tasks, such as secondary structure prediction and contact map prediction.

Original: We make the following key observations:
Revised: We make the following key observations:

Original: (i) The AlphaFold-trained Evoformer produces representations that are useful for both structure and function prediction, as shown on two structure prediction tasks, two function annotation tasks [48], and two fitness score prediction tasks [28].
Revised: (i) The AlphaFold-trained Evoformer produces representations that are useful for both structure and function prediction, as shown on two structure prediction tasks, two function annotation tasks [48], and two ﬁtness score prediction tasks [28].

Original: (ii) Evoformer representations are useful for both protein-level and residue-level prediction tasks.
Revised: (ii) Evoformer representations are useful for both protein-level and residue-level prediction tasks.

Original: (iii) Evoformer is superior to ESM-1b and MSA-Transformer for structure prediction and novel miniprotein stability prediction, but inferior to ESM-1b on other function prediction tasks.
Revised: (iii) Evoformer is superior to ESM-1b and MSA-Transformer for structure prediction and novel miniprotein stability prediction, but inferior to ESM-1b on other function prediction tasks.

Original: It performs poorly for zero-shot fitness prediction tasks [27] compared to ESM-1b and MSA-Transformer.
Revised: It performs poorly for zero-shot ﬁtness prediction tasks [27] compared to ESM-1b and MSA-Transformer.

Original: (iv) Evolution-aware PLMs are superior to evolution-free ESM-1b model only in the structure prediction tasks, but in general, are worse than ESM-1b in most function prediction tasks.
Revised: (iv) Evolution-aware PLMs are superior to evolution-free ESM-1b model only in the structure prediction tasks, but in general, are worse than ESM-1b in most function prediction tasks.

Original: (v) Like structure prediction, evolution-aware PLMs are also sensitive to the amount of MSAs when predicting protein functions.
Revised: (v) Like structure prediction, evolution-aware PLMs are also sensitive to the amount of MSAs when predicting protein functions.

Original: In addition, their performance using ESM-1b-constructed MSAs as input is comparable to the performance using MSAs generated by Jackhmmer [22] or HHblits [32].
Revised: In addition, their performance using ESM-1b-constructed MSAs as input is comparable to the performance using MSAs generated by Jackhmmer [22] or HHblits [32].

Original:  Related Work
Revised:  Related Work

Original: Protein language models The volume of protein data has exploded over the last decade with the advancement of new DNA sequencing technologies.
Revised: Protein language models The volume of protein data has exploded over the last decade with the advancement of new DNA sequencing technologies.

Original: Early work learned protein representations with LSTMs [18, 4, 1].
Revised: Early work learned protein representations with LSTMs [18, 4, 1].

Original: Recently, with the advent of large transformer models in natural language processing (NLP), large PLMs using the Transformer [43] architecture and BERT [12] denoising
Revised: Recently, with the advent of large transformer models in natural language processing (NLP), large PLMs using the Transformer [43] architecture and BERT [12] denoising

Original: [Table]
Revised: [Table]

Original: task have been widely adopted [35, 28, 13].
Revised: task have been widely adopted [35, 28, 13].

Original: * PLMs trained on large sequence databases have been successfully applied for various protein related tasks, including secondary structure prediction [28, 18, 35, 13], contact prediction [29, 35], 3D structure prediction [23, 2], annotation prediction [7], signal peptide prediction [40], intracellular localization prediction [41], protein-protein interaction prediction [14], and fitness prediction [1, 10, 19, 20, 27].
Revised: * PLMs trained on large sequence databases have been successfully applied for various protein related tasks, including secondary structure prediction [28, 18, 35, 13], contact prediction [29, 35], 3D structure prediction [23, 2], annotation prediction [7], signal peptide prediction [40], intracellular localization prediction [41], protein-protein interaction prediction [14], and ﬁtness prediction [1, 10, 19, 20, 27].

Original: ESM-1b
Revised: ESM-1b

Original: [35] found that residue-residue contacts can be recovered from the learned representation, identifying the close relationship between
Revised: [35] found that residue-residue contacts can be recovered from the learned representation, identifying the close relationship between

Original: Transformer attention and biological features.
Revised: Transformer attention and biological features.

Original: Following this, [44] and [29] further studied the interpretability of the attention map as contact map.
Revised: Following this, [44] and [29] further studied the interpretability of the attention map as contact map.

Original: ProtTrans [13] benchmarked a variety of BERTlike PLMs, including TAPE, ESM-1b, ProtTrans, and MSA-Transformer.
Revised: ProtTrans [13] benchmarked a variety of BERTlike PLMs, including TAPE, ESM-1b, ProtTrans, and MSA-Transformer.

Original: Since ProtTrans with the
Revised: Since ProtTrans with the

Original: BERT architecture does not exceed ESM-1b, we choose to compare ESM-1b, MSA-Transformer, and Evoformer for this study.
Revised: BERT architecture does not exceed ESM-1b, we choose to compare ESM-1b, MSA-Transformer, and Evoformer for this study.

Original: Table 1 and Figure 1 provide more details about the three models.
Revised: Table 1 and Figure 1 provide more details about the three models.

Original: Structure, function, and fitness prediction A protein’s primary structure (i.e. AA sequence) determines its 3D structure, which in turn determines its functional properties.
Revised: Structure, function, and ﬁtness prediction A protein’s primary structure (i.e. AA sequence) determines its 3D structure, which in turn determines its functional properties.

Original: This relationship underlies the success of PLMs, which infer protein structure and function from the raw sequence.
Revised: This relationship underlies the success of PLMs, which infer protein structure and function from the raw sequence.

Original: Secondary structure (SS) prediction [38] is an easier task than predicting the 2D contact map [45] and 3D structure [23].
Revised: Secondary structure (SS) prediction [38] is an easier task than predicting the 2D contact map [45] and 3D structure [23].

Original: MSA-Transformer and ESM-1b can make accurate unsupervised contact predictions, which are an important input for the 3D structure generation.
Revised: MSA-Transformer and ESM-1b can make accurate unsupervised contact predictions, which are an important input for the 3D structure generation.

Original: While remarkable progress has been made in structure prediction, it is unknown whether this
Revised: While remarkable progress has been made in structure prediction, it is unknown whether this

Original: AlphaFold-triggered revolution transfers to other tasks, in particular function prediction.
Revised: AlphaFold-triggered revolution transfers to other tasks, in particular function prediction.

Original: Protein function is a broad term that refers to any biological or biochemical roles in organisms.
Revised: Protein function is a broad term that refers to any biological or biochemical roles in organisms.

Original: In this paper, we focus on functional annotation prediction (classification task) [48, 6] and fitness prediction (including both regression [10, 28] and zero-shot prediction [19, 27] tasks), as shown in Figure 2.
Revised: In this paper, we focus on functional annotation prediction (classiﬁcation task) [48, 6] and ﬁtness prediction (including both regression [10, 28] and zero-shot prediction [19, 27] tasks), as shown in Figure 2.

Original: Compared with annotation prediction, fitness prediction is more challenging since (1) there are routinely few or no laboratory labels for supervision; (2) protein AA sequences are highly similar given the same wild-type protein.
Revised: Compared with annotation prediction, ﬁtness prediction is more challenging since (1) there are routinely few or no laboratory labels for supervision; (2) protein AA sequences are highly similar given the same wild-type protein.

Original: [Figure]
Revised: [Figure]

Original:  Preliminaries 3.1 Tasks and datasets
Revised:  Preliminaries 3.1 Tasks and datasets

Original: We evaluate models on two structure prediction tasks:
Revised: We evaluate models on two structure prediction tasks:

Original: Secondary structure (SS): This is a residue-level sequence-to-sequence task where each residue x i of a protein sequence x = { x 1 , x 2 , ..., x L } is mapped to a label
Revised: Secondary structure (SS): This is a residue-level sequence-to-sequence task where each residue x i of a protein sequence x = { x 1 , x 2 , ..., x L } is mapped to a label

Original: y i corresponding to one of eight secondary structure tasks y i ∈ { G, H, ..., C } [28].
Revised: y i corresponding to one of eight secondary structure tasks y i 2 { G, H, ..., C } [28].

Original: SS prediction examines the degree to which a PLM learns local structure.
Revised: SS prediction examines the degree to which a PLM learns local structure.

Original: Contacts:
Revised: Contacts:

Original: For a given protein structure, two residues are considered to be in contact if their C β carbons are within 8Å. We evaluate on pairs that are more than 6 positions apart in the primary structure [28].
Revised: For a given protein structure, two residues are considered to be in contact if their C  carbons are within 8Å. We evaluate on pairs that are more than 6 positions apart in the primary structure [28].

Original: We measure the results using Precision@ L , which stands for the precision for the top- L pairs with the highest predicted contact probability [47].
Revised: We measure the results using Precision@ L , which stands for the precision for the top- L pairs with the highest predicted contact probability [47].

Original: L is the length of the protein sequence.
Revised: L is the length of the protein sequence.

Original: For both contacts and secondary structure, we use the dataset in [35] which is constructed from
Revised: For both contacts and secondary structure, we use the dataset in [35] which is constructed from

Original: SCOPe [15], and use the suggested split as the training and testing sets (see Table 2).
Revised: SCOPe [15], and use the suggested split as the training and testing sets (see Table 2).

Original: One concern is that the dataset used here has been trained by AlphaFold as they come from the Protein Data
Revised: One concern is that the dataset used here has been trained by AlphaFold as they come from the Protein Data

Original: Bank (PDB) [3].
Revised: Bank (PDB) [3].

Original: Hence, we investigate 48 additional proteins, which were collected from CAMEO † (Continuous Automated Model EvaluatiOn) with ‘hard’ category from 2021-08-28 to 2022-04-30.
Revised: Hence, we investigate 48 additional proteins, which were collected from CAMEO † (Continuous Automated Model EvaluatiOn) with ‘hard’ category from 2021-08-28 to 2022-04-30.

Original: We also evaluate on two function (annotation) classification tasks:
Revised: We also evaluate on two function (annotation) classiﬁcation tasks:

Original: Metal ion binding (MIB):
Revised: Metal ion binding (MIB):

Original: This is a binary classification task, where a PLM with a new classification layer is used to determine whether there are metal ion–binding sites in the protein.
Revised: This is a binary classiﬁcation task, where a PLM with a new classiﬁcation layer is used to determine whether there are metal ion–binding sites in the protein.

Original: The dataset is also collected from PDB with annotation as metal ion binding.
Revised: The dataset is also collected from PDB with annotation as metal ion binding.

Original: We randomly sample the same amount of proteins from the database as the negative class.
Revised: We randomly sample the same amount of proteins from the database as the negative class.

Original: Antibiotic resistance (ABR):
Revised: Antibiotic resistance (ABR):

Original: This is a multi-class classification task, where a PLM need to correctly determine which class of antibiotic a protein degrades.
Revised: This is a multi-class classiﬁcation task, where a PLM need to correctly determine which class of antibiotic a protein degrades.

Original: We construct the dataset from CARD [26] which contains 19 different antibiotic types (see Appendix A.5 for details)
Revised: We construct the dataset from CARD [26] which contains 19 different antibiotic types (see Appendix A.5 for details).

Original: And three fitness prediction tasks.
Revised: And three ﬁtness prediction tasks.

Original: Unlike functional annotation prediction, protein sequences in this task are all from the same wild-type with a small number of mutated residues.
Revised: Unlike functional annotation prediction, protein sequences in this task are all from the same wild-type with a small number of mutated residues.

Original: Stability: This is a protein-level regression task that predicts the protease concentration at which a protein can maintain its fold [36].
Revised: Stability: This is a protein-level regression task that predicts the protease concentration at which a protein can maintain its fold [36].

Original: We use the data splits from TAPE.
Revised: We use the data splits from TAPE.

Original: Fluorescence:
Revised: Fluorescence:

Original: This is also a protein-level regression task, predicting the log-fluorescence intensity of the protein sequence [37].
Revised: This is also a protein-level regression task, predicting the log-ﬂuorescence intensity of the protein sequence [37].

Original: We use the data splits from TAPE.
Revised: We use the data splits from TAPE.

Original: Zero-shot mutation effect prediction: This is a protein-level prediction task by comparing the difference between likelihoods assigned to the mutated residue and the likelihoods assigned to the wild-type (see [27] for details).
Revised: Zero-shot mutation effect prediction: This is a protein-level prediction task by comparing the difference between likelihoods assigned to the mutated residue and the likelihoods assigned to the wild-type (see [27] for details).

Original: We evaluate five protein mutation datasets from DeepSequence [34].
Revised: We evaluate ﬁve protein mutation datasets from DeepSequence [34].

Original: Only single point mutation data is considered in this sub-task.
Revised: Only single point mutation data is considered in this sub-task.

Original: [Table]
Revised: [Table]

Original: 3.2 Methods
Revised: 3.2 Methods

Original: For supervised function prediction tasks, we remove the original classification layer for ESM-1b and MSA-Transformer, and the Structure module for AlphaFold.
Revised: For supervised function prediction tasks, we remove the original classiﬁcation layer for ESM-1b and MSA-Transformer, and the Structure module for AlphaFold.

Original: These remaining parts are called protein representations.
Revised: These remaining parts are called protein representations.

Original: We then add a linear layer on top of these representation models to perform new classification or regression tasks.
Revised: We then add a linear layer on top of these representation models to perform new classiﬁcation or regression tasks.

Original: We adopt the standard fine-tuning strategy by fine-tuning all parameters using Adamw optimizer with 1e-5 as learning rate.
Revised: We adopt the standard ﬁne-tuning strategy by ﬁne-tuning all parameters using Adamw optimizer with 1e-5 as learning rate.

Original: For the zero-shot fitness prediction task, no training is needed.
Revised: For the zero-shot ﬁtness prediction task, no training is needed.

Original: Instead, we obtain the softmax output at the corresponding mutation site in each protein sequence.
Revised: Instead, we obtain the softmax output at the corresponding mutation site in each protein sequence.

Original: The probability value is regarded as the fitness value for each amino acid following [19, 27].
Revised: The probability value is regarded as the ﬁtness value for each amino acid following [19, 27].

Original: As for structure predictions, we follow the same practice in [28].
Revised: As for structure predictions, we follow the same practice in [28].

Original: Other details can be seen in Appendix A.1.
Revised: Other details can be seen in Appendix A.1.

Original:  Results 4.1 Structure Prediction
Revised:  Results 4.1 Structure Prediction

Original: In this section, we examine the structure representation ability of the three PLMs models discussed above.
Revised: In this section, we examine the structure representation ability of the three PLMs models discussed above.

Original: In general, the secondary structure and contact prediction tasks do not have significant practical values since there is already highly accurate 3D structure data by AlphaFold.
Revised: In general, the secondary structure and contact prediction tasks do not have signiﬁcant practical values since there is already highly accurate 3D structure data by AlphaFold.

Original: The purpose instead is to provide a reference for function prediction, given that some tasks have similar formulation.
Revised: The purpose instead is to provide a reference for function prediction, given that some tasks have similar formulation.

Original: Table 3 and 9 (Appendix A.4) show the results of SS prediction and contact prediction, respectively.
Revised: Table 3 and 9 (Appendix A.4) show the results of SS prediction and contact prediction, respectively.

Original: For SS prediction, we consider two settings: pre-trained parameters and training from scratch (i.e., with random initialization).
Revised: For SS prediction, we consider two settings: pre-trained parameters and training from scratch (i.e., with random initialization).

Original: For Evoformer, pre-training means the training of AlphaFold.
Revised: For Evoformer, pre-training means the training of AlphaFold.

Original: For contact prediction, all PLMs have been already pre-trained since there is no meaningful contact map without pre-training.
Revised: For contact prediction, all PLMs have been already pre-trained since there is no meaningful contact map without pre-training.

Original: First, we can easily observe that with pre-training, Evoformer performs the best in both tasks.
Revised: First, we can easily observe that with pre-training, Evoformer performs the best in both tasks.

Original: Particularly, Evoformer outperforms ESM-1b and MSA-Transformer by a large margin for contact prediction with over 94% accuracy on Precision@ L (see Table 9).
Revised: Particularly, Evoformer outperforms ESM-1b and MSA-Transformer by a large margin for contact prediction with over 94% accuracy on Precision@ L (see Table 9).

Original: By contrast, it does not perform as accurately as the contact prediction task for SS prediction, which improves ESM-1b by 11.6% and MSA-Transformer by 5%.
Revised: By contrast, it does not perform as accurately as the contact prediction task for SS prediction, which improves ESM-1b by 11.6% and MSA-Transformer by 5%.

Original: This is reasonable since SS prediction is different from 3D structure prediction from the machine learning perspective.
Revised: This is reasonable since SS prediction is different from 3D structure prediction from the machine learning perspective.

Original: Even though Evoformer has strong structure representation ability, it may not work well by just adding a linear classification layer.
Revised: Even though Evoformer has strong structure representation ability, it may not work well by just adding a linear classiﬁcation layer.

Original: This suggests that a more complex structure module for SS prediction is necessary for higher accuracy.
Revised: This suggests that a more complex structure module for SS prediction is necessary for higher accuracy.

Original: In comparison, since contact information can be directly extracted from the pairwise distance map, it is not surprising that Evoformer shows superb results in this task.
Revised: In comparison, since contact information can be directly extracted from the pairwise distance map, it is not surprising that Evoformer shows superb results in this task.

Original: Thereby, we have also demonstrated results ofnew proteins in Figure 3, where these proteins have not been trained by AlphaFold.
Revised: Thereby, we have also demonstrated results ofnew proteins in Figure 3, where these proteins have not been trained by AlphaFold.

Original: In general, they are consistent with our above analysis.
Revised: In general, they are consistent with our above analysis.

Original: Second, we observe that pre-training is important: all PLMs are remarkably improved with around 40% improvement for ESM-1b and 28% for Evoformer.
Revised: Second, we observe that pre-training is important: all PLMs are remarkably improved with around 40% improvement for ESM-1b and 28% for Evoformer.

Original: Interestingly, we note that Evoformer performs worse than MSA-Transformer by training from scratch.
Revised: Interestingly, we note that Evoformer performs worse than MSA-Transformer by training from scratch.

Original: We conjecture that Evoformer is much harder to be trained without good initialization as its network architecture is more complex and deeper than MSA-Transformer.
Revised: We conjecture that Evoformer is much harder to be trained without good initialization as its network architecture is more complex and deeper than MSA-Transformer.

Original: Finally, for structure prediction, evolution-aware models are largely better than evolution-free ESM-1b.
Revised: Finally, for structure prediction, evolution-aware models are largely better than evolution-free ESM-1b.

Original: This is consistent with biological intuition and previous observation, such as in [30].
Revised: This is consistent with biological intuition and previous observation, such as in [30].

Original: [Table]
Revised: [Table]

Original: 4.2 Supervised Function Prediction
Revised: 4.2 Supervised Function Prediction

Original: AlphaFold predicts very accurate protein structures.
Revised: AlphaFold predicts very accurate protein structures.

Original: However, it remains unknown whether its core representation module Evoformer can prediction function.
Revised: However, it remains unknown whether its core representation module Evoformer can prediction function.

Original: Similarly, we investigate MSATransformer’s ability to predict protein function, which is unknown either.
Revised: Similarly, we investigate MSATransformer’s ability to predict protein function, which is unknown either.

Original: Note that, unlike annotation prediction, evolution-aware Evoformer and MSA-Transformer only take single sequences (rather than MSAs) as input since all protein variants share the same MSAs for the fitness prediction task.
Revised: Note that, unlike annotation prediction, evolution-aware Evoformer and MSA-Transformer only take single sequences (rather than MSAs) as input since all protein variants share the same MSAs for the ﬁtness prediction task.

Original: The results of annotation prediction (protein-level classification) and fitness predictions (protein-level regression) are listed in Table 4 and 5 respectively.
Revised: The results of annotation prediction (protein-level classiﬁcation) and ﬁtness predictions (protein-level regression) are listed in Table 4 and 5 respectively.

Original: We use classification accuracy [28] for evaluating annotation prediction and Spearman rank correlation [27] for evaluating fitness prediction.
Revised: We use classiﬁcation accuracy [28] for evaluating annotation prediction and Spearman rank correlation [27] for evaluating ﬁtness prediction.

Original: First, we find that for annotation prediction, ESM-1b, the evolution-free PLM, exceeds the two evolution-aware
Revised: First, we ﬁnd that for annotation prediction, ESM-1b, the evolution-free PLM, exceeds the two evolution-aware

Original: PLMs, although opposite observations are made for predicting structures.
Revised: PLMs, although opposite observations are made for predicting structures.

Original: Specifically, pre-trained
Revised: Speciﬁcally, pre-trained

Original: ESM-1b achieves 5.8% and 17.4% improvements over pre-trained Evoformer and MSA-Transformer.
Revised: ESM-1b achieves 5.8% and 17.4% improvements over pre-trained Evoformer and MSA-Transformer.

Original: This is a bit surprising since the biological intuition is that protein functional properties are mediated by structures.
Revised: This is a bit surprising since the biological intuition is that protein functional properties are mediated by structures.

Original: An important conclusion we reached here is that better structure PLMs do not mean they have a better representation for predicting function.
Revised: An important conclusion we reached here is that better structure PLMs do not mean they have a better representation for predicting function.

Original: Second, all pre-trained PLMs perform very well on the ABR task, a multi-class protein classification task.
Revised: Second, all pre-trained PLMs perform very well on the ABR task, a multi-class protein classiﬁcation task.

Original: Despite that, we can still observe an obvious improvement between training-from-scratch and pre-training.
Revised: Despite that, we can still observe an obvious improvement between training-from-scratch and pre-training.

Original: Similar observation brought from pre-training can be obviously seen from all other function prediction tasks.
Revised: Similar observation brought from pre-training can be obviously seen from all other function prediction tasks.

Original: This suggests that both supervised and unsupervised pre-training on
Revised: This suggests that both supervised and unsupervised pre-training on

Original: PLMs are very useful to obtain protein function representations.
Revised: PLMs are very useful to obtain protein function representations.

Original: Third, we observe that the three pre-trained PLMs in general show better results than the one-hot [28] and ResNet [28, 17] baselines in Table 5.
Revised: Third, we observe that the three pre-trained PLMs in general show better results than the one-hot [28] and ResNet [28, 17] baselines in Table 5.

Original: This observation is new since neither Evoformer nor
Revised: This observation is new since neither Evoformer nor

Original: MSA-Transformer have been investigated for supervised fitness prediction.
Revised: MSA-Transformer have been investigated for supervised ﬁtness prediction.

Original: In more detail, ESM-1b performs the best on the fluorescence task, whereas Evoformer is the best on the stability task.
Revised: In more detail, ESM-1b performs the best on the ﬂuorescence task, whereas Evoformer is the best on the stability task.

Original: This may be because protein stability has a closer relationship with protein structure, or because sequence-only pretraining does not generalize as well as structural pretraining to de novo miniproteins.
Revised: This may be because protein stability has a closer relationship with protein structure, or because sequence-only pretraining does not generalize as well as structural pretraining to de novo miniproteins.

Original: 4.3 Zero-shot Mutation effects Prediction
Revised: 4.3 Zero-shot Mutation effects Prediction

Original: Recent work revealed that PLMs are strong zero-shot learners in predicting potential viral mutations [19, 20].
Revised: Recent work revealed that PLMs are strong zero-shot learners in predicting potential viral mutations [19, 20].

Original: We are interested in evaluating Evoformer’s masked-residue reconstruction ability in such a zero-shot setting given that it also has a BERT-like masked token prediction loss.
Revised: We are interested in evaluating Evoformer’s masked-residue reconstruction ability in such a zero-shot setting given that it also has a BERT-like masked token prediction loss.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Figure 4 shows the results of all PLMs.
Revised: Figure 4 shows the results of all PLMs.

Original: First, we observe that ESM-1b and ESM-1v [27] yield the best results
Revised: First, we observe that ESM-1b and ESM-1v [27] yield the best results

Original: [Table]
Revised: [Table]

Original: and both of them are evolution-free models.
Revised: and both of them are evolution-free models.

Original: ESM-1v shares an architecture with ESM-1b, but is an ensemble of 5 models trained on UniRef90 instead of UniRef50.
Revised: ESM-1v shares an architecture with ESM-1b, but is an ensemble of 5 models trained on UniRef90 instead of UniRef50.

Original: MSA-Transformer in general performs well except on BLAT_ECOLX_Ostermeier2014.
Revised: MSA-Transformer in general performs well except on BLAT_ECOLX_Ostermeier2014.

Original: We notice that MSA-Transformer is highly affected by the quality and sequence identity thresholds of MSAs.
Revised: We notice that MSA-Transformer is highly affected by the quality and sequence identity thresholds of MSAs.

Original: For our experiments, we use the default setting by HHblits [33] searching from BFD ‡ database (see Appendix A.3)
Revised: For our experiments, we use the default setting by HHblits [33] searching from BFD ‡ database (see Appendix A.3)

Original: [Table]
Revised: [Table]

Original: The most surprising results are for Evoformer, which performs poorly in this task on all five datasets.
Revised: The most surprising results are for Evoformer, which performs poorly in this task on all ﬁve datasets.

Original: To examine this cause, we present results of another three models, namely ESM1-85M (released by [35]), ESM1b-88M (af2_data) and MSA-Transformer (af2_data) (see model descriptions in Figure 4).
Revised: To examine this cause, we present results of another three models, namely ESM1-85M (released by [35]), ESM1b-88M (af2_data) and MSA-Transformer (af2_data) (see model descriptions in Figure 4).

Original: As shown, ESM1-85M performs much better than ESM1b-88M, which indicates that the accuracy of  zero-shot fitness prediction task is largely affected by the size of the pre-training dataset.
Revised: As shown, ESM1-85M performs much better than ESM1b-88M, which indicates that the accuracy of the zero-shot ﬁtness prediction task is largely affected by the size of the pre-training dataset.

Original: This may explain why Evoformer performs so poorly in this task.
Revised: This may explain why Evoformer performs so poorly in this task.

Original: Beyond this, there is another key difference between these models.
Revised: Beyond this, there is another key difference between these models.

Original: Evoformer or AlphaFold were trained with five different loss functions, among which four are structure-related loss and only one auxiliary loss is the masked token reconstruction (MTR) loss.
Revised: Evoformer or AlphaFold were trained with ﬁve different loss functions, among which four are structure-related loss and only one auxiliary loss is the masked token reconstruction (MTR) loss.

Original: By contrast, MSA-Transformer and ESM-1b were trained with only the MTR loss.
Revised: By contrast, MSA-Transformer and ESM-1b were trained with only the MTR loss.

Original: This may also be a reason for explaining the worse results of Evoformer.
Revised: This may also be a reason for explaining the worse results of Evoformer.

Original: Answer for Q(i):
Revised: Answer for Q(i):

Original: Yes .
Revised: Yes .

Original: The answer is provided in Table 4 and 5.
Revised: The answer is provided in Table 4 and 5.

Original: From these results, we can see that Evoformer with pre-trained parameters largely outperforms the non-pretrained version in four different function prediction tasks.
Revised: From these results, we can see that Evoformer with pre-trained parameters largely outperforms the non-pretrained version in four different function prediction tasks.

Original: Together with its structure representation ability, we conclude that parameters learned by AlphaFold are general-purpose and useful to various structure and function prediction tasks.
Revised: Together with its structure representation ability, we conclude that parameters learned by AlphaFold are general-purpose and useful to various structure and function prediction tasks.

Original: Answer for Q(ii): No .
Revised: Answer for Q(ii): No .

Original: The answer is provided in Table 3, 9, 4, 5 and Figure 3.
Revised: The answer is provided in Table 3, 9, 4, 5 and Figure 3.

Original: These results show that Evoformer indeed outperforms MSA-Transformer and ESM-1b when predicting structures, but it does not obviously outperform ESM-1b when predicting function.
Revised: These results show that Evoformer indeed outperforms MSA-Transformer and ESM-1b when predicting structures, but it does not obviously outperform ESM-1b when predicting function.

Original: In particular, it does not work at all in the zero-shot mutation effect prediction task.
Revised: In particular, it does not work at all in the zero-shot mutation effect prediction task.

Original: Accordingly, we can conclude that the AlphaFold-triggered revolution for structure prediction cannot be directly transferred to function predictions.
Revised: Accordingly, we can conclude that the AlphaFold-triggered revolution for structure prediction cannot be directly transferred to function predictions.

Original: ESM-1b is still the SOTA when predicting protein functions.
Revised: ESM-1b is still the SOTA when predicting protein functions.

Original: 4.4 Effect of MSA
Revised: 4.4 Effect of MSA

Original: Here, we want to understand the influence of MSAs on evolution-aware PLMs.
Revised: Here, we want to understand the inﬂuence of MSAs on evolution-aware PLMs.

Original: Further, we explore the relationship between evolution-aware and -free models.
Revised: Further, we explore the relationship between evolution-aware and -free models.

Original: As mentioned in Section 4.2, evolutionaware PLMs take only a single sequence as input for the supervised fitness prediction tasks, so we do not consider the affect of MSAs for these two tasks.
Revised: As mentioned in Section 4.2, evolutionaware PLMs take only a single sequence as input for the supervised ﬁtness prediction tasks, so we do not consider the affect of MSAs for these two tasks.

Original: [Table]
Revised: [Table]

Original: We report results on Table 6 and Figure 5.
Revised: We report results on Table 6 and Figure 5.

Original: The MSA setting in Table 6 is consistent for both model training and inference.
Revised: The MSA setting in Table 6 is consistent for both model training and inference.

Original: For Figure 5, we only consider the inference stage because it is a zero-shot task.
Revised: For Figure 5, we only consider the inference stage because it is a zero-shot task.

Original: Clearly, we observe that without MSAs, Evoformer, and MSA-Transformer both yield much worse results (with or without pre-training).
Revised: Clearly, we observe that without MSAs, Evoformer, and MSA-Transformer both yield much worse results (with or without pre-training).

Original: Similarly, as shown in Figure 5, MSA-Transformer is also affected by the number of MSAs.
Revised: Similarly, as shown in Figure 5, MSA-Transformer is also affected by the number of MSAs.

Original: In general, it performs much worse with few MSAs.
Revised: In general, it performs much worse with few MSAs.

Original: These results show that ( Answer for Q(iii) )
Revised: These results show that ( Answer for Q(iii) )

Original: MSA data is a crucial input for evolution-aware PLMs, for not only the structure prediction but also the function prediction tasks.
Revised: MSA data is a crucial input for evolution-aware PLMs, for not only the structure prediction but also the function prediction tasks.

Original: 4.5 Relationship between Evolution-aware & -free PLMs
Revised: 4.5 Relationship between Evolution-aware & -free PLMs

Original: Evolution-aware PLMs require MSA data as input for both training and inference.
Revised: Evolution-aware PLMs require MSA data as input for both training and inference.

Original: However, database search using HHblits [33] or Jackhmmer [22] can take up to 10 minutes for a 1000-residue protein.
Revised: However, database search using HHblits [33] or Jackhmmer [22] can take up to 10 minutes for a 1000-residue protein.

Original: On the other hand, given the powerful function representation of ESM-1b, we design a fast homology retrieval framework using Siamese ESM-1b, following [31].
Revised: On the other hand, given the powerful function representation of ESM-1b, we design a fast homology retrieval framework using Siamese ESM-1b, following [31].

Original: Figure 6 is the schematic of the ESM1b-constructed MSA generating and serving process, called ESM-MSA (see Appendix A.2).
Revised: Figure 6 is the schematic of the ESM1b-constructed MSA generating and serving process, called ESM-MSA (see Appendix A.2).

Original: [Table]
Revised: [Table]

Original: Table 7 shows the results by using different MSAs for Evoformer and MSA-Transformer.
Revised: Table 7 shows the results by using different MSAs for Evoformer and MSA-Transformer.

Original: MSAs generated by ESM-1b are comparable to those found by Jackhmmer (or HHblits) while being much faster (see Appendix A.3.2), suggesting that ( Answer for Q(iii) )
Revised: MSAs generated by ESM-1b are comparable to those found by Jackhmmer (or HHblits) while being much faster (see Appendix A.3.2), suggesting that ( Answer for Q(iii) )

Original: the function representation learned by evolution-free ESM-1b can be used to construct MSAs for evolution-aware MSATransformer and Evoformer.
Revised: the function representation learned by evolution-free ESM-1b can be used to construct MSAs for evolution-aware MSATransformer and Evoformer.

Original:  Conclusions and Limitations
Revised:  Conclusions and Limitations

Original: In the paper, we have presented empirical studies to characterize three successful PLMs.
Revised: In the paper, we have presented empirical studies to characterize three successful PLMs.

Original: In particular, we focus on the function representation capacity of Evoformer.
Revised: In particular, we focus on the function representation capacity of Evoformer.

Original: We draw several important conclusions: (i) Evoformer encodes not only structure, but also various protein functional properties; it provides an alternative choice when predicting protein functions, especially the stability prediction task; (ii) despite that AlphaFold dominates the structure prediction tasks, Evoformer is not yet able to substitute ESM-1b and MSA-Transformer when predicting protein functions; (iii) the functional predictability of evolution-aware PLMs is also influenced by protein MSAs.
Revised: We draw several important conclusions: (i) Evoformer encodes not only structure, but also various protein functional properties; it provides an alternative choice when predicting protein functions, especially the stability prediction task; (ii) despite that AlphaFold dominates the structure prediction tasks, Evoformer is not yet able to substitute ESM-1b and MSA-Transformer when predicting protein functions; (iii) the functional predictability of evolution-aware PLMs is also inﬂuenced by protein MSAs.

Original: Interestingly, evolution-free ESM-1b can generate sufficiently-accurate MSAs for evolution-aware PLMs with much higher efficiency.
Revised: Interestingly, evolution-free ESM-1b can generate sufﬁciently-accurate MSAs for evolution-aware PLMs with much higher efﬁciency.

Original: Our work points out both strengths and weakness of three SOTA PLMs, which are potentially useful to many downstream biological tasks, including but not limited to protein engineering [8], cancer early detection [16], and drug discovery [24].
Revised: Our work points out both strengths and weakness of three SOTA PLMs, which are potentially useful to many downstream biological tasks, including but not limited to protein engineering [8], cancer early detection [16], and drug discovery [24].

Original: An important limitation is that we probe the representation capacity of these PLMs by only adding a linear head on top — a common practice in the deep learning community.
Revised: An important limitation is that we probe the representation capacity of these PLMs by only adding a linear head on top — a common practice in the deep learning community.

Original: While this verifies our findings, we believe such a linear layer is not sufficiently expressive to achieve SOTA results.
Revised: While this veriﬁes our ﬁndings, we believe such a linear layer is not sufﬁciently expressive to achieve SOTA results.

Original: That is the reason why Evoformer with a eight-layer Structure module achieves much higher 3D structure accuracy but with a linear layer it achieves less than 80% classification accuracy for the secondary structure prediction task.
Revised: That is the reason why Evoformer with a eight-layer Structure module achieves much higher 3D structure accuracy but with a linear layer it achieves less than 80% classiﬁcation accuracy for the secondary structure prediction task.

Original: In addition, we do not fully disentangle the effects of the Evoformer architecture, training data, and loss function when comparing it to MSA-Transformer.
Revised: In addition, we do not fully disentangle the effects of the Evoformer architecture, training data, and loss function when comparing it to MSA-Transformer.

Original: Therefore, an important direction is to develop more advanced functional modules for higher prediction accuracy.
Revised: Therefore, an important direction is to develop more advanced functional modules for higher prediction accuracy.

Original: Acknowledgement
Revised: Acknowledgement

Original: This work is supported byspecial funding from the Westlake Center of Synthetic Biology and 
Revised: This work is supported by the National Key Research and Development Program of China (No.

Revised: 2022ZD0115100), the National Natural Science Foundation of China (No. U21A20427), the Westlake

Original:  Integrated Bioengineering (WE-SynBio), the National Natural Science Foundation of China (No.
Revised: Center of Synthetic Biology and Integrated Bioengineering (WE-SynBio), the Research Center for

Original: U21A20427), and the Research Center for Industries of the Future (No. WU2022C030).
Revised: Industries of the Future (No. WU2022C030), and Key Research Project of Zhejiang Lab (No. 2022PG0AC02).

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] (c) Did you discuss any potential negative societal impacts of your work?
Revised: [Yes] (c) Did you discuss any potential negative societal impacts of your work?

Original: [No] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [No] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] (b) Did you include complete proofs of all theoretical results?
Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 3.
Revised: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] We will release code and datasets for reproducibility.
Revised: [Yes] We will release code and datasets for reproducibility.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] See Sections A.4 and A. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: [Yes] See Sections A.4 and A. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] In figure 5, we report the effect of different MSAs on the results.
Revised: [Yes] In ﬁgure 5, we report the effect of different MSAs on the results.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] See appendix A.6. 4.
Revised: [Yes] See appendix A.6. 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [Yes] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [Yes] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Original: [N/A] 5.
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?[N/A] A Appendix A.1 Methodology Details
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: For protein-level prediction tasks, we extract representation features from the [CLS] token of the last layer of ESM-1b and MSA-Transformer and add a new linear layer on top of it.

Original: Since there is no [CLS] token appended in AlphaFold, we average all residue embeddings from the last layer and add the same linear layer on top.

Original: For residue-level supervised prediction task (i.e. SS prediction), we add a linear layer on top of each residue embedding the last layer in all three PLMs.

Original: We train these models for each non-zero-shot task by fine-tuning all parameters, including both the new linear layer and the backbone representation model.

Original: For contact prediction, we perform experiments differently for the three models given that the pairwise distance matrix of Evoformer can be directly extracted.

Original: For MSA-Transformer and ESM-1b, we follow the [28], extracting and normalizing the attention map from all hidden layers and then training a linear layer on these 2D maps and performing the regression task.

Original: For zero-shot fitness prediction, MSAs are searched in the BFD dataset [23] using Jackhmmer [22] with the default parameters.

Original: For annotation prediction tasks, we search MSAs from Uniref90 [42] database with Jackhmmer.

Original: For SS and contact prediction, we use the MSAs provided by [35].

Original: MSAs are not used in the two supervised fitness prediction tasks because all sequences are highly similar with only a few positions different.

Original: We report all results of Evoformer without template to avoid information leaking for the structure tasks.

Original: A.2 ESM-MSA details A.2.1 Training set construction

Original: ESM-MSA is essentially a two-tower-based network following sentence-BERT [31] where each tower is represented by the ESM-1b encoder.

Original: Unlike sentence-BERT, we have developed an effective negative sampling method to choose more informative negative protein sequences rather than perform random sampling.

Original: The schematic of ESM-MSA is shown in Figure 6 and more details are given below:

Original: We first collect high-quality homologous sequence datasets.

Original: We use the public trRosetta training set § as the ground truth data in this work.

Original: Define H =

Original: { h 1 , h 2 , ..., h n } as the entire protein sequence data set, where h i represents the aligned homologous sequences for protein i , usually called an MSA.

Original: Define h i = { q 1 , q 2 , ..., q n } , where q i is an individual sequence in a same homologous family.

Original: Then define D neg = { d 1 , d 2 , ..., d n } as the data for negative sampling, where d i also denotes an individual protein sequence.

Original: We use Uniclust30 ¶ for sampling non-homologous sequences, which includemillion individual protein sequences.

Original: We then construct a training set including homologous sequence pairs and non-homologous sequence pairs to train the model.

Original: Specifically, for a protein sequence q i belonging to set h i , we randomly sample its homologous sequence q pos ∈

Original: h i from the same set and calculate the biological identity s ∈ [0 , 1] between the two sequences.

Original: Finally, we construct a positive sample pair in the training set p pos .

Original: Similarly, for the original sequence q i ∈

Original: h

Original: i , we sample a sequence q neg ∈ D from the Uniclust30 database as a negative sample(non-homologous sequence), thus constructing a negative sample pair P neg .

Original: We finally obtain the set of positive pairs

Original: S pos =

Original: { p pos 1 , p pos 2 , ...., p posn } , and the set of negative pairs S neg =

Original: { p neg 1 , p neg 2 , ...., p negm } , and then obtain the complete training set S = S pos (cid:83) S neg .

Original: A.2.2 Dynamic negative sampling

Original: In this paper, we use dynamic negative sampling at the training stage, a.k.a.

Original: hard negative sampling [46].

Original: That is, instead of randomly sampling sequences, we continuously select samples that are more difficult for the model to distinguish.

Original: By this way, the model convergence speed is greatly accelerated and the performance of the model is largely improved.

Original: Specifically, for the newly initialized model, we randomly select a batch of samples from the database as negative samples for training.

Original: After that, in every new round, for each sequence q i , we randomly sample n sequences from the database and calculate the Euclidean distance between the original sequence and sampled sequences by the model.

Original: We select the sampled sequence with the closest Euclidean distance to the original sequence representation as the negative sample.

Original: Since the closest Euclidean distance means that the model is not able to distinguish it as negative.

Original: To reduce the computational cost, we use a sequence pooling approach, i.e., we first randomly sample N sequences( N ≫ n ) from the database as a sub-database, and then perform subsequent sampling operations on such sub-database after the corresponding scores are computed by the model.

Original: A.2.3 Objective Function

Original: For the positive sample pair, the loss function is defined as follows:

Original: [Equation]

Original: where p is the original sequence, q is the homologous sequence of p , and s is the biological identity of the two sequences.

Original: The definition is based on the hypothesis that the greater the biological sequence identity of the homologous sequences, the closer the Euclidean distance between the two should be through the representations given by the model.

Original: This means that between homologous sequences, the Euclidean distance also varies according to the biological sequence identity, and thus the model can obtain better generalization performance.

Original: Therefore, we used the mean square error loss function.

Original: Because s ∈ [0 , 1] , setting the distance threshold to 1 not only conforms to Occam’s razor principle [9], but also fits well with the loss function.

Original: For the negative sample pair, the loss function is defined as follows:

Original: [Equation]

Original: where z ( p, q ) =

Original: e dist ( p, 1) e dist ( p,q ) + e t , t is distance threshold.

Original: Actually, this is Cross Entropy function.

Original: For non-homologous sequences, we want the model to keep increasing the Euclidean distance between p and q , so it is reasonable to use the distance threshold t as a reference for the Cross Entropy, so that the Euclidean distance of the representations keeps moving away from the distance threshold.

Original: [Table]

Original: A.2.4 MSA Retrieval and Alignment

Original: First, we use the embedded vector of ’CLS’ in ESM-1b (trained by the above approach) as the protein representation.

Original: We calculate all protein sequences in the database.

Original: Then we use Faiss [21], the library for quick embedding searching, to retrieve homologous sequences.

Original: Specifically, we calculate the

Original: Euclidean distance between sequence embeddings and keep those under the threshold as homologous to the query protein.

Original: One we obtain a set of homologous sequences, we use Famsa[11] to efficiently align them and output an a3m format file.

Original: A.3 HHblits setting A.3.1 Search the BFD database HHblits setting

Original: n_iter: int = 3, e_value: float = 0.001, maxseq: int = 1000000, realign_max: int = 100000, maxfilt: int = 100000, min_prefilter_hits: int = 1000, all_seqs: bool =

Original: False, p: 20, z: int = 500.

Original: A.3.2 Compare with the running time of ESM-MSA vs HHblits n_iter: int = 1, e_value: float = 0.0001, maxseq: int = 1000, realign_max: int = 100000, maxfilt: int = 100000, min_prefilter_hits: int = 1000, all_seqs: bool =

Original: False, p: 20, z: int = 500.

Original: [Table]
Revised: [N/A]

Original: A.4 Contact map Results on SCOPe

Original: See Table 9 for details.

Original: A.5 Antibiotic Resistance dataset detail

Original: The Antibiotic Resistance dataset is derived from experimentally verified bacterial antibiotic resistance proteins from the Comprehensive Antibiotic Resistance Database (CARD), and redundant sequences with 100% identity are removed using the CD-HIT tool [25].

Original: Finally, a total of 2602 protein sequences from 19 antibiotic classes are constructed for functional classification and analysis.

Original: A.6 GPU cluster

Original: All our experiments are performed on the NVIDIA A40 with 48G GPU memory.

Original: A.7 Remote Homology Detection (Evolutionary Understanding Task)

Original: Here, we add the results of the remote homology detection task, which has exactly the same training and testing set in TAPE.

Original: As shown in Table 10, we can make the same observations as in the annotation prediction task.

Original: First, the performance of all three PLMs is substantially improved by pre-training, which shows the representation ability of them; Second, ESM-1b performs better than

Original: MSA-Transformer and Evoformer.

Original: Our results are consistent with those in TAPE.

Original: It is worth noting that although the homology detection task measures a model’s ability to detect structural similarity, it is essentially formulated as a protein-level annotation or classification task, like the MIB and ABR tasks in this paper.

Original: By comparison, the typical structural prediction tasks, including the SS prediction, contact prediction, and 3D structure prediction, are atom- or residue-level prediction, where residue is often represented by the C α or C β atom.

Original: [Table]