Original: Exploring evolution-based & -free protein language models as protein function predictors
Revised: Exploring evolution-based & -free protein language models as protein function predictors

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Large-scale Protein Language Models (PLMs) have improved performance inprotein prediction tasks, ranging from 3D structure prediction to various functionpredictions.
Revised: Large-scale Protein Language Models (PLMs) have improved performance inprotein prediction tasks, ranging from 3D structure prediction to various functionpredictions.

Original: In particular, AlphaFold [23], a ground-breaking AI system, couldpotentially reshape structural biology.
Revised: In particular, AlphaFold [23], a ground-breaking AI system, couldpotentially reshape structural biology.

Original: However, the utility of the PLM module in
Revised: However, the utility of the PLM module in

Original: AlphaFold, Evoformer, has not been explored beyond structure prediction.
Revised: AlphaFold, Evoformer, has not been explored beyond structure prediction.

Original: In thispaper, we investigate the representation ability of three popular PLMs: ESM-1b(single sequence) [35], MSA-Transformer (multiple sequence alignment) [30] and
Revised: In thispaper, we investigate the representation ability of three popular PLMs: ESM-1b(single sequence) [35], MSA-Transformer (multiple sequence alignment) [30] and

Original: Evoformer (structural), with a special focus on Evoformer.
Revised: Evoformer (structural), with a special focus on Evoformer.

Original: Speciﬁcally, we aimto answer the following key questions: (i) Does the Evoformer trained as partof AlphaFold produce representations amenable to predicting protein function?
Revised: Speciﬁcally, we aimto answer the following key questions: (i) Does the Evoformer trained as partof AlphaFold produce representations amenable to predicting protein function?

Original: (ii)
Revised: (ii)

Original: If yes, can Evoformer replace ESM-1b and MSA-Transformer?
Revised: If yes, can Evoformer replace ESM-1b and MSA-Transformer?

Original: (iii)
Revised: (iii)

Original: How
Revised: How

Original: much do these PLMs rely on evolution-related protein data?
Revised: much do these PLMs rely on evolution-related protein data?

Original: In this regard, arethey complementary to each other?
Revised: In this regard, arethey complementary to each other?

Original: We compare these models by empirical studyalong with new insights and conclusions.
Revised: We compare these models by empirical studyalong with new insights and conclusions.

Original: Finally, we release code and datasets forreproducibility.
Revised: Finally, we release code and datasets forreproducibility.

Original:  Introduction
Revised:  Introduction

Original: Proteins perform the majority of biological activities.
Revised: Proteins perform the majority of biological activities.

Original: It is, therefore, crucial to decipher the mechanisms underlying their structural and functional properties.
Revised: It is, therefore, crucial to decipher the mechanisms underlying their structural and functional properties.

Original: The canonical sequence-structure-functionrelationship enables the success of sequence-based machine learning methods that infer protein structure and function from amino acid (AA) sequence.
Revised: The canonical sequence-structure-functionrelationship enables the success of sequence-based machine learning methods that infer protein structure and function from amino acid (AA) sequence.

Original: Large-scale protein language models (PLMs) withself-supervised pretraining on tens of millions to billions of proteins (PLMs) [28, 35, 13, 5] are thecurent state-of-the-art in predicting function and ﬁtness from sequences.
Revised: Large-scale protein language models (PLMs) withself-supervised pretraining on tens of millions to billions of proteins (PLMs) [28, 35, 13, 5] are thecurent state-of-the-art in predicting function and ﬁtness from sequences.

Original: Meanwhile
Revised: Meanwhile

Original: AlphaFold [23], trained on experimental 3D protein structures from the Protein Data
Revised: AlphaFold [23], trained on experimental 3D protein structures from the Protein Data

Original: Bank (PDB) [39] can approach the resolution of experimental structures for most protein sequences.
Revised: Bank (PDB) [39] can approach the resolution of experimental structures for most protein sequences.

Original: Its multiple sequence alignment representation module, Evoformer, combines new deep learningmachinery, a PLM residue reconstruction task, and structural supervision in the form of a distogram.
Revised: Its multiple sequence alignment representation module, Evoformer, combines new deep learningmachinery, a PLM residue reconstruction task, and structural supervision in the form of a distogram.

Original: Like MSA-Transformer [30], Evoformer takes a family of evolutionarily-related and aligned proteinsequences as input, in contrast to PLMs such as ESM-1b [35] and TAPE [28], which only takeindividual protein sequences.
Revised: Like MSA-Transformer [30], Evoformer takes a family of evolutionarily-related and aligned proteinsequences as input, in contrast to PLMs such as ESM-1b [35] and TAPE [28], which only takeindividual protein sequences.

Original: Thus, for short, we refer to the former two models as evolution-based
Revised: Thus, for short, we refer to the former two models as evolution-based

Original: PLMs and the latter two as evolution-free PLMs.
Revised: PLMs and the latter two as evolution-free PLMs.

Original: Despite the remarkable success of AlphaFold in predicting structure from sequence, it is unknownwhether its Evoformer module can be applied to other problems, in particular predicting proteinfunction and ﬁtness.
Revised: Despite the remarkable success of AlphaFold in predicting structure from sequence, it is unknownwhether its Evoformer module can be applied to other problems, in particular predicting proteinfunction and ﬁtness.

Original: Deciphering AlphaFold rather than treating it as a black box is beneﬁcial to both artiﬁcial intelligence and biology communities.
Revised: Deciphering AlphaFold rather than treating it as a black box is beneﬁcial to both artiﬁcial intelligence and biology communities.

Original: Therefore, we attempt to answer the following keyquestions.
Revised: Therefore, we attempt to answer the following keyquestions.

Original: Q(i): Does the Evoformer trained during AlphaFold training learn general-purpose proteinrepresentations that can be used for various function prediction tasks?
Revised: Q(i): Does the Evoformer trained during AlphaFold training learn general-purpose proteinrepresentations that can be used for various function prediction tasks?

Original: Unlike ESM-1b and MSATransformer, Evoformer is trained with supervision from 3D structures.
Revised: Unlike ESM-1b and MSATransformer, Evoformer is trained with supervision from 3D structures.

Original: In addition, the second partof AlphaFold, the Structure Module, built on top of the 48 Evoformer blocks, is much more complexand deeper than the traditional (linear) classiﬁcation head used in ESM-1b and MSA-Transformer.
Revised: In addition, the second partof AlphaFold, the Structure Module, built on top of the 48 Evoformer blocks, is much more complexand deeper than the traditional (linear) classiﬁcation head used in ESM-1b and MSA-Transformer.

Original: These differences make the function representation ability of Evoformer an open question.
Revised: These differences make the function representation ability of Evoformer an open question.

Original: Q(ii):
Revised: Q(ii):

Original: If Evoformer’s representation is general, does it outperform ESM-1b and MSATransformer on downstream tasks?
Revised: If Evoformer’s representation is general, does it outperform ESM-1b and MSATransformer on downstream tasks?

Original: While these three models are trained with different parametersizes and datasets (see Table 1), we believe the comparison results are still valuable because they arecurrently the most advanced PLMs.
Revised: While these three models are trained with different parametersizes and datasets (see Table 1), we believe the comparison results are still valuable because they arecurrently the most advanced PLMs.

Original: Training these large models from scratch is out of the reach ofmost academic research groups due to the compute and cost involved.
Revised: Training these large models from scratch is out of the reach ofmost academic research groups due to the compute and cost involved.

Original: In addition, we also investigatethe utility of ESM-1b and MSA-Transformer on function prediction tasks.
Revised: In addition, we also investigatethe utility of ESM-1b and MSA-Transformer on function prediction tasks.

Original: Q(iii): How much does the performance of evolution-based PLMs rely on the input MSAs?
Revised: Q(iii): How much does the performance of evolution-based PLMs rely on the input MSAs?

Original: Can evolution-free PLMs assist evolution-based PLMs in terms of MSA construction?
Revised: Can evolution-free PLMs assist evolution-based PLMs in terms of MSA construction?

Original: Both
Revised: Both

Original: Evoformer and MSA-Transformer take sets of aligned sequences (MSAs) as input.
Revised: Evoformer and MSA-Transformer take sets of aligned sequences (MSAs) as input.

Original: We investigatethe effect of MSA quality and depth on function prediction.
Revised: We investigatethe effect of MSA quality and depth on function prediction.

Original: On the other side, ESM-1b can simplyformulate MSA construction as the remote homology detection task.
Revised: On the other side, ESM-1b can simplyformulate MSA construction as the remote homology detection task.

Original: Then, it is interesting to knowwhether ESM1b-constructed MSAs can be used as inputs to Evoformer and MSA-Transformer.
Revised: Then, it is interesting to knowwhether ESM1b-constructed MSAs can be used as inputs to Evoformer and MSA-Transformer.

Original: We address the above questions through comprehensive empirical studies on a variety of structureand function prediction tasks.
Revised: We address the above questions through comprehensive empirical studies on a variety of structureand function prediction tasks.

Original: Note although AlphaFold (i.e. Evoformer + Structure Module) canaccurately predict protein 3D structures, we investigate the ability of Evoformer to perform otherstructure prediction tasks, such as secondary structure prediction and contact map prediction.
Revised: Note although AlphaFold (i.e. Evoformer + Structure Module) canaccurately predict protein 3D structures, we investigate the ability of Evoformer to perform otherstructure prediction tasks, such as secondary structure prediction and contact map prediction.

Original: Wemake the following key observations:
Revised: Wemake the following key observations:

Original: (i) The AlphaFold-trained Evoformer produces representations that are useful for both structure and
Revised: (i) The AlphaFold-trained Evoformer produces representations that are useful for both structure and

Original: function prediction, as shown on two structure prediction tasks, two function annotation tasks [48],and two ﬁtness score prediction tasks [28].(ii) Evoformer representations are useful for both protein-level and residue-level prediction tasks.(iii)
Revised: function prediction, as shown on two structure prediction tasks, two function annotation tasks [48],and two ﬁtness score prediction tasks [28].(ii) Evoformer representations are useful for both protein-level and residue-level prediction tasks.

Original:  Evoformer is superior to ESM-1b and MSA-Transformer for structure prediction and novel
Revised: (iii) Evoformer is superior to ESM-1b and MSA-Transformer for structure prediction and novel

Original: miniprotein stability prediction, but inferior to ESM-1b on other function prediction tasks.
Revised: miniprotein stability prediction, but inferior to ESM-1b on other function prediction tasks.

Original: It performspoorly for zero-shot ﬁtness prediction tasks [27] compared to ESM-1b and MSA-Transformer.(iv)
Revised: It performspoorly for zero-shot ﬁtness prediction tasks [27] compared to ESM-1b and MSA-Transformer.(iv)

Original: Evolution-based PLMs are superior to evolution-free ESM-1b model only in the structure
Revised: Evolution-based PLMs are superior to evolution-free ESM-1b model only in the structure

Original: prediction tasks, but in general, are worse than ESM-1b in most function prediction tasks anddatasets.
Revised: prediction tasks, but in general, are worse than ESM-1b in most function prediction tasks anddatasets.

Original: (v) Consistent with structure prediction, evolution-based PLMs are also sensitive to the amount ofMSAs when predicting protein function.
Revised: (v) Consistent with structure prediction, evolution-based PLMs are also sensitive to the amount ofMSAs when predicting protein function.

Original: In addition, their performance using ESM-1b-constructedMSAs as input is comparable to the performance using MSAs generated by Jackhmmer [22] orHHblits [32].
Revised: In addition, their performance using ESM-1b-constructedMSAs as input is comparable to the performance using MSAs generated by Jackhmmer [22] orHHblits [32].

Original:  Related Work
Revised:  Related Work

Original: Protein language models The volume of protein data has exploded over the last decade with theadvancement of new DNA sequencing technologies.
Revised: Protein language models The volume of protein data has exploded over the last decade with theadvancement of new DNA sequencing technologies.

Original: Early work learned protein structural representation with a bidirectional LSTM (biLSTM) [18, 4, 1].
Revised: Early work learned protein representationswith LSTMs [18, 4, 1].

Original: Recently, with the advent of large transformermodels in natural language processing (NLP), large PLMs using the Transformer [43] architecture and BERT [12] denoising task have been widely adopted [35, 28, 13].
Revised: Recently, with the advent of large transformer models in natural languageprocessing (NLP), large PLMs using the Transformer [43] architecture and BERT [12] denoisingtask have been widely adopted [35, 28, 13].

Original: 1 PLMs trained on large sequencedatabases have been successfully applied for various protein related tasks, including secondary struc- ture prediction [28, 18, 35, 13], contact prediction [29, 35], 3D structure prediction [23, 2], annotationprediction [7], signal peptide prediction [40], intracellular localization prediction [41], protein-proteininteraction prediction [14]
Revised: 1 PLMs trained on large sequence databases have beensuccessfully applied for various protein related tasks, including secondary structure prediction [28, 18, 35, 13], contact prediction [29, 35], 3D structure prediction [23, 2], annotation prediction [7],signal peptide prediction [40], intracellular localization prediction [41], protein-protein interactionprediction [14]

Original: [Table]
Revised: [Table]

Original: and ﬁtness prediction [1, 10, 19, 20, 27].
Revised: and ﬁtness prediction [1, 10, 19, 20, 27].

Original: ESM-1b
Revised: ESM-1b

Original: [35] found that residueresidue contacts can be recovered from the learned representation, identifying the close relationshipbetween Transformer attention and biological features.
Revised: [35] found that residue-residuecontacts can be recovered from the learned representation, identifying the close relationship between Transformer attention and biological features.

Original: Following this, [44] and [29] further studiedthe interpretability of the attention map as contact map.
Revised: Following this, [44] and [29] further studied theinterpretability of the attention map as contact map.

Original: ProtTrans [13] benchmarked a variety of
Revised: ProtTrans [13] benchmarked a variety of

Original: BERT-like PLMs, including TAPE, ESM-1b,
Revised: BERT-like PLMs, including TAPE, ESM-1b,

Original: ProtTrans
Revised: ProtTrans

Original: [Table]
Revised: [Table]

Original: and MSA-Transformer.
Revised: and MSA-Transformer.

Original: Since ProtTrans withthe BERT architecture does not exceed ESM-1b, we choose to compare ESM-1b, MSA-Transformer,and Evoformer for this study.
Revised: Since ProtTrans withthe BERT architecture does not exceed ESM-1b, we choose to compare ESM-1b, MSA-Transformer,and Evoformer for this study.

Original: Table 1 and Figure 1 provide more details about the three models
Revised: Table 1 and Figure 1 provide more details about the three models

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]

Original: Evoformer

Original: MSA
Revised: MSA

Original: Pair
Revised: Pair

Original: Evoformer (cid:11)(cid:23)(cid:27)(cid:3)(cid:69)(cid:79)(cid:82)(cid:70)(cid:78)(cid:86)(cid:12)
Revised: Evoformer (cid:11)(cid:23)(cid:27)(cid:3)(cid:69)(cid:79)(cid:82)(cid:70)(cid:78)(cid:86)(cid:12)

Original: Structure module (cid:11)(cid:27)(cid:3)(cid:69)(cid:79)(cid:82)(cid:70)(cid:78)(cid:86)(cid:12)
Revised: Structure module (cid:11)(cid:27)(cid:3)(cid:69)(cid:79)(cid:82)(cid:70)(cid:78)(cid:86)(cid:12)

Original: [Figure]
Revised: d. Alphafold

Original: Pair MSA
Revised: Pair MSA

Original: Seqence (cid:74)(cid:72)(cid:81)(cid:72)(cid:87)(cid:76)(cid:70)(cid:71)(cid:68)(cid:87)(cid:68)(cid:69)(cid:68)(cid:86)(cid:72)(cid:86)(cid:72)(cid:68)(cid:85)(cid:70)(cid:75)
Revised: Seqence (cid:74)(cid:72)(cid:81)(cid:72)(cid:87)(cid:76)(cid:70)(cid:71)(cid:68)(cid:87)(cid:68)(cid:69)(cid:68)(cid:86)(cid:72)(cid:86)(cid:72)(cid:68)(cid:85)(cid:70)(cid:75)

Original: (cid:83)(cid:68)(cid:76)(cid:85)(cid:76)(cid:81)(cid:74)
Revised: (cid:83)(cid:68)(cid:76)(cid:85)(cid:76)(cid:81)(cid:74)

Original: Structure, function, and ﬁtness prediction A protein’s primary structure (i.e. AA sequence)determines its 3D structure, which in turn determines its functional properties.
Revised: Structure, function, and ﬁtness prediction A protein’s primary structure (i.e. AA sequence)determines its 3D structure, which in turn determines its functional properties.

Original: This relationshipunderlies the success of PLMs, which infer protein structure and function from the raw sequence.
Revised: This relationshipunderlies the success of PLMs, which infer protein structure and function from the raw sequence.

Original: Secondary structure (SS) prediction [38] is an easier task than predicting the 2D contact map [45]and 3D structure [23].
Revised: Secondary structure (SS) prediction [38] is an easier task than predicting the 2D contact map [45]and 3D structure [23].

Original: MSA-Transformer and ESM-1b can make accurate unsupervised contactpredictions, which are an important input for the 3D structure generation.
Revised: MSA-Transformer and ESM-1b can make accurate unsupervised contactpredictions, which are an important input for the 3D structure generation.

Original: While remarkable progress has been made in structure prediction, it is unknown whether this
Revised: While remarkable progress has been made in structure prediction, it is unknown whether this

Original: AlphaFold-triggered revolution transfers to other tasks, in particular function prediction.
Revised: AlphaFold-triggered revolution transfers to other tasks, in particular function prediction.

Original: Proteinfunction is a broad term that refers to any biological or biochemical roles in organisms.
Revised: Proteinfunction is a broad term that refers to any biological or biochemical roles in organisms.

Original: In thispaper, we focus on functional annotation prediction (classiﬁcation task) [48, 6] and ﬁtness prediction(including both regression [10, 28] and zero-shot prediction [19, 27] tasks), as shown in Figure 2.
Revised: In thispaper, we focus on functional annotation prediction (classiﬁcation task) [48, 6] and ﬁtness prediction(including both regression [10, 28] and zero-shot prediction [19, 27] tasks), as shown in Figure 2.

Original: Compared with annotation prediction, ﬁtness prediction is more challenging since (1) there are routinely few or no laboratory labels for supervision; (2) protein AA sequences are highly similar given the same wild-type protein.
Revised: Compared with annotation prediction, ﬁtness prediction is more challenging since (1) there areroutinely few or no laboratory labels for supervision; (2) protein AA sequences are highly similargiven the same wild-type protein.

Original:  Preliminaries
Revised:  Preliminaries

Original: [Figure]
Revised: [Figure]

Original: 3.1 Tasks and datasets
Revised: 3.1 Tasks and datasets

Original: [Figure]
Revised: [Figure]

Original: We evaluate models on two structure prediction tasks:
Revised: We evaluate models on two structure prediction tasks:

Original: Secondary structure (SS): This is a residue-level sequence-to-sequence task where each
Revised: Secondary structure (SS): This is a residue-level sequence-to-sequence task where each

Original: residue x i of a protein sequence x = { x 1 , x 2 , ..., x L } is mapped to a label y i correspondingto one of eight secondary structure tasks y
Revised: residue x i of a protein sequence x = { x 1 , x 2 , ..., x L } is mapped to a label y i correspondingto one of eight secondary structure tasks y

Original: i 2 { G, H, ..., C } [28].
Revised: i 2 { G, H, ..., C } [28].

Original: SS prediction examines
Revised: SS prediction examines

Original: the degree to which a PLM learns local structure.
Revised: the degree to which a PLM learns local structure.

Original: Contacts:
Revised: Contacts:

Original: For a given protein structure, two residues are considered to be in contact if their
Revised: For a given protein structure, two residues are considered to be in contact if their

Original: C  carbons are within 8Å. We evaluate on pairs that are more than 6 positions apart in the
Revised: C  carbons are within 8Å. We evaluate on pairs that are more than 6 positions apart in the

Original: primary structure [28].
Revised: primary structure [28].

Original: We measure the results using Precision@ L , which stands for the
Revised: We measure the results using Precision@ L , which stands for the

Original: precision for the top- L pairs with the highest predicted contact probability [47].
Revised: precision for the top- L pairs with the highest predicted contact probability [47].

Original: L is the
Revised: L is the

Original: length of the protein sequence.
Revised: length of the protein sequence.

Original: For both contacts and secondary structure, we use the dataset in [35] which is constructed from
Revised: For both contacts and secondary structure, we use the dataset in [35] which is constructed from

Original: SCOPe [15], and use the suggested split as the training and testing sets (see Table 2).
Revised: SCOPe [15], and use the suggested split as the training and testing sets (see Table 2).

Original: One concernis that the dataset used here has been trained by AlphaFold as they come from the Protein Data
Revised: One concernis that the dataset used here has been trained by AlphaFold as they come from the Protein Data

Original: Bank (PDB) [3].
Revised: Bank (PDB) [3].

Original: Hence, we investigate 48 additional proteins, which were collected from CAMEO 2(Continuous Automated Model EvaluatiOn) with ‘hard’ category from 2021-08-28 to 2022-04-30.
Revised: Hence, we investigate 48 additional proteins, which were collected from CAMEO 2(Continuous Automated Model EvaluatiOn) with ‘hard’ category from 2021-08-28 to 2022-04-30.

Original: We also evaluate on two function (annotation) classiﬁcation tasks:
Revised: We also evaluate on two function (annotation) classiﬁcation tasks:

Original: Metal ion binding (MIB):
Revised: Metal ion binding (MIB):

Original: This is a binary classiﬁcation task, where a PLM with a newclassiﬁcation layer is used to determine whether there are metal ion–binding sites in theprotein.
Revised: This is a binary classiﬁcation task, where a PLM with a newclassiﬁcation layer is used to determine whether there are metal ion–binding sites in theprotein.

Original: The dataset is also collected from PDB with annotation as metal ion binding.
Revised: The dataset is also collected from PDB with annotation as metal ion binding.

Original: Werandomly sample the same amount of proteins from the database as the negative class.
Revised: Werandomly sample the same amount of proteins from the database as the negative class.

Original: Antibiotic resistance (ABR):
Revised: Antibiotic resistance (ABR):

Original: This is a multi-class classiﬁcation task, where a PLM need tocorrectly determine which class of antibiotic a protein degrades.
Revised: This is a multi-class classiﬁcation task, where a PLM need tocorrectly determine which class of antibiotic a protein degrades.

Original: We construct the datasetfrom CARD [
Revised: We construct the datasetfrom CARD [

Original: 26] which contains 19 different antibiotic types (see Appendix A.5 for details).
Revised: 26] which contains 19 different antibiotic types (see Appendix A.5 for details).

Original: And three ﬁtness prediction tasks.
Revised: And three ﬁtness prediction tasks.

Original: Unlike functional annotation prediction, protein sequences in thistask are all from the same wild-type with a small number of mutated residues.
Revised: Unlike functional annotation prediction, protein sequences in thistask are all from the same wild-type with a small number of mutated residues.

Original: Stability: This is a protein-level regression task that predicts the protease concentration atwhich a protein can maintain its fold [36].
Revised: Stability: This is a protein-level regression task that predicts the protease concentration atwhich a protein can maintain its fold [36].

Original: We use the data splits from TAPE.
Revised: We use the data splits from TAPE.

Original: Fluorescence: This is also a protein-level regression task, predicting the log-ﬂuorescence
Revised: Fluorescence: This is also a protein-level regression task, predicting the log-ﬂuorescence

Original: intensity of the protein sequence [37].
Revised: intensity of the protein sequence [37].

Original: We use the data splits from TAPE.
Revised: We use the data splits from TAPE.

Original: Zero-shot mutation effect prediction: This is a protein-level prediction task by comparing the
Revised: Zero-shot mutation effect prediction: This is a protein-level prediction task by comparing the

Original: difference between likelihoods assigned to the mutated residue and the likelihoods assigned
Revised: difference between likelihoods assigned to the mutated residue and the likelihoods assigned

Original: to the wild-type (see [27] for details).
Revised: to the wild-type (see [27] for details).

Original: We evaluate ﬁve protein mutation datasets from
Revised: We evaluate ﬁve protein mutation datasets from

Original: DeepSequence [34]
Revised: DeepSequence [34]

Original: Only single point mutation data is considered in this sub-task
Revised: Only single point mutation data is considered in this sub-task

Original: 3.2 Methods
Revised: 3.2 Methods

Original: For supervised function prediction tasks, we remove the original classiﬁcation layer for ESM-1band MSA-Transformer, and the Structure module for AlphaFold.
Revised: For supervised function prediction tasks, we remove the original classiﬁcation layer for ESM-1band MSA-Transformer, and the Structure module for AlphaFold.

Original: These remaining parts are calledprotein representations.
Revised: These remaining parts are calledprotein representations.

Original: We then add a linear layer on top of these representation models to performnew classiﬁcation or regression tasks.
Revised: We then add a linear layer on top of these representation models to performnew classiﬁcation or regression tasks.

Original: We adopt the standard ﬁne-tuning strategy by ﬁne-tuning allparameters using Adamw optimizer with 1e-5 as learning rate.
Revised: We adopt the standard ﬁne-tuning strategy by ﬁne-tuning allparameters using Adamw optimizer with 1e-5 as learning rate.

Original: For the zero-shot ﬁtness predictiontask, no training is needed.
Revised: For the zero-shot ﬁtness predictiontask, no training is needed.

Original: Instead, we obtain the softmax output at the corresponding mutation sitein each protein sequence.
Revised: Instead, we obtain the softmax output at the corresponding mutation sitein each protein sequence.

Original: The probability value is regarded as the ﬁtness value for each amino acid
Revised: The probability value is regarded as the ﬁtness value for each amino acid

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: TAPE
Revised: TAPE

Original: [Table]
Revised: [Table]

Original: following [19, 27].
Revised: following [19, 27].

Original: As for structure predictions, we follow the same practice in [28].
Revised: As for structure predictions, we follow the same practice in [28].

Original: Other detailscan be seen in Appendix A.1.
Revised: Other detailscan be seen in Appendix A.1.

Original:  Results
Revised:  Results

Original: 4.1 Structure Prediction
Revised: 4.1 Structure Prediction

Original: In this section, we examine the structure representation ability of the three PLMs models discussedabove.
Revised: In this section, we examine the structure representation ability of the three PLMs models discussedabove.

Original: In general, the secondary structure and contact prediction tasks do not have signiﬁcant practicalvalues since there is already highly accurate 3D structure data by AlphaFold.
Revised: In general, the secondary structure and contact prediction tasks do not have signiﬁcant practicalvalues since there is already highly accurate 3D structure data by AlphaFold.

Original: The purpose instead isto provide a reference for function prediction, given that some tasks have similar formulation.
Revised: The purpose instead isto provide a reference for function prediction, given that some tasks have similar formulation.

Original: Table 3 and 9 (Appendix A.4) show the results of SS prediction and contact prediction, respectively.
Revised: Table 3 and 9 (Appendix A.4) show the results of SS prediction and contact prediction, respectively.

Original: For SS prediction, we consider two settings: pre-trained parameters and training from scratch (i.e.,with random initialization).
Revised: For SS prediction, we consider two settings: pre-trained parameters and training from scratch (i.e.,with random initialization).

Original: For Evoformer, pre-training means the training of AlphaFold.
Revised: For Evoformer, pre-training means the training of AlphaFold.

Original: For contactprediction, all PLMs have been already pre-trained since there is no meaningful contact map withoutpre-training.
Revised: For contactprediction, all PLMs have been already pre-trained since there is no meaningful contact map withoutpre-training.

Original: [Table]
Revised: [Table]

Original: First, we can easily observe that with pre-training, Evoformer performs the best in both tasks.
Revised: First, we can easily observe that with pre-training, Evoformer performs the best in both tasks.

Original: Particularly, Evoformer outperforms ESM-1b and MSA-Transformer by a large margin for contactprediction with over 94% accuracy on Precision@ L (see Table 9).
Revised: Particularly, Evoformer outperforms ESM-1b and MSA-Transformer by a large margin for contactprediction with over 94% accuracy on Precision@ L (see Table 9).

Original: By contrast, it does not perform asaccurately as the contact prediction task for SS prediction, which improves ESM-1b by 11.6% and MSA-Transformer by 5%.
Revised: By contrast, it does not performas accurately as the contact prediction task for SS prediction, which improves ESM-1b by 11.6%and MSA-Transformer by 5%.

Original: This is reasonable since SS prediction isvery different from 3D structureprediction.
Revised: This is reasonable since SS prediction is different from 3D structureprediction from the machine learning perspective.

Original: Even though Evoformer has strong structure representation ability, it may not work wellby just adding a linear classiﬁcation layer.
Revised: Even though Evoformer has strong structure representation ability, it may not work well by just adding a linear classiﬁcation layer.

Original: This suggests that a more complex structure module for SSprediction is necessary for higher accuracy.
Revised: This suggests that amore complex structure module for SS prediction is necessary for higher accuracy.

Original: In comparison, since contact information can be directlyextracted from the pairwise distance map, it is not surprising that Evoformer shows superb results inthis task.
Revised: In comparison,since contact information can be directly extracted from the pairwise distance map, it is not surprisingthat Evoformer shows superb results in this task.

Original: Thereby, we have also demonstrated results of 48 new proteins in Figure 3, where theseproteins have not been trained by AlphaFold.
Revised: Thereby, we have also demonstrated results of 48new proteins in Figure 3, where these proteins have not been trained by AlphaFold.

Original: In general, they are consistent with our above analysis.
Revised: In general, theyare consistent with our above analysis.

Original: Second, we observe that pre-training is important: all PLMs are remarkably improved with around40% improvement for ESM-1b and 28% for Evoformer.
Revised: Second, we observe that pre-training is important: all PLMs are remarkably improved with around40% improvement for ESM-1b and 28% for Evoformer.

Original: Interestingly, we note that Evoformerperforms worse than MSA-Transformer by training from scratch.
Revised: Interestingly, we note that Evoformerperforms worse than MSA-Transformer by training from scratch.

Original: We conjecture that Evoformer ismuch harder to be trained without good initialization as its network architecture is more complex anddeeper than MSA-Transformer.
Revised: We conjecture that Evoformer ismuch harder to be trained without good initialization as its network architecture is more complex anddeeper than MSA-Transformer.

Original: Finally, for structure prediction, evolution-based models are largely better than evolution-free ESM-1b.
Revised: Finally, for structure prediction, evolution-based models are largely better than evolution-free ESM-1b.

Original: This is consistent with biological intuition and previous observation, such as in [30].
Revised: This is consistent with biological intuition and previous observation, such as in [30].

Original: 4.2 Supervised Function Prediction
Revised: 4.2 Supervised Function Prediction

Original: AlphaFold predicts very accurate protein structures.
Revised: AlphaFold predicts very accurate protein structures.

Original: However, it remains unknown whether itscore representation module Evoformer can prediction function.
Revised: However, it remains unknown whether itscore representation module Evoformer can prediction function.

Original: Similarly, we investigate MSATransformer’s ability to predict protein function, which is unknown either.
Revised: Similarly, we investigate MSATransformer’s ability to predict protein function, which is unknown either.

Original: Note that, unlike annotationprediction, evolution-based Evoformer and MSA-Transformer only take single sequences (rather than
Revised: Note that, unlike annotationprediction, evolution-based Evoformer and MSA-Transformer only take single sequences (rather than

Original: MSAs) as input since all protein variants share the same MSAs for the ﬁtness prediction task.
Revised: MSAs) as input since all protein variants share the same MSAs for the ﬁtness prediction task.

Original: The results of annotation prediction (protein-level classiﬁcation) and ﬁtness predictions (protein-levelregression) are listed in Table 4 and 5 respectively.
Revised: The results of annotation prediction (protein-level classiﬁcation) and ﬁtness predictions (protein-levelregression) are listed in Table 4 and 5 respectively.

Original: We use classiﬁcation accuracy [28] for evaluatingannotation prediction and Spearman rank correlation [27] for evaluating ﬁtness prediction.
Revised: We use classiﬁcation accuracy [28] for evaluatingannotation prediction and Spearman rank correlation [27] for evaluating ﬁtness prediction.

Original: First, weﬁnd that for annotation prediction, ESM-1b, the evolution-free PLM, exceeds the two evolution-based
Revised: First, weﬁnd that for annotation prediction, ESM-1b, the evolution-free PLM, exceeds the two evolution-based

Original: PLMs, although opposite observations are made for predicting structures.
Revised: PLMs, although opposite observations are made for predicting structures.

Original: Speciﬁcally, pre-trained
Revised: Speciﬁcally, pre-trained

Original: ESM-1b achieves 5.8% and 17.4% improvements over pre-trained Evoformer and MSA-Transformer.
Revised: ESM-1b achieves 5.8% and 17.4% improvements over pre-trained Evoformer and MSA-Transformer.

Original: This is a bit surprising since the biological intuition is that protein functional properties are mediatedby structures.
Revised: This is a bit surprising since the biological intuition is that protein functional properties are mediatedby structures.

Original: An important conclusion we reached here is that better structure PLMs do not meanthey have a better representation for predicting function.
Revised: An important conclusion we reached here is that better structure PLMs do not meanthey have a better representation for predicting function.

Original: Second, all pre-trained PLMs perform very well on the ABR task, a multi-class protein classiﬁcationtask.
Revised: Second, all pre-trained PLMs perform very well on the ABR task, a multi-class protein classiﬁcationtask.

Original: Despite that, we can still observe an obvious improvement between training-from-scratch andpre-training.
Revised: Despite that, we can still observe an obvious improvement between training-from-scratch andpre-training.

Original: Similar observation brought from pre-training can be obviously seen from all otherfunction prediction tasks.
Revised: Similar observation brought from pre-training can be obviously seen from all otherfunction prediction tasks.

Original: This suggests that both supervised and unsupervised pre-training on
Revised: This suggests that both supervised and unsupervised pre-training on

Original: PLMs are very useful to obtain protein function representations.
Revised: PLMs are very useful to obtain protein function representations.

Original: Third, we observe that the three pre-trained PLMs in general show better results than the one-hot [28]and ResNet [28, 17] baselines in Table 5.
Revised: Third, we observe that the three pre-trained PLMs in general show better results than the one-hot [28]and ResNet [28, 17] baselines in Table 5.

Original: This observation is new since neither Evoformer nor
Revised: This observation is new since neither Evoformer nor

Original: MSA-Transformer have been investigated for supervised ﬁtness prediction.
Revised: MSA-Transformer have been investigated for supervised ﬁtness prediction.

Original: In more detail, ESM-1bperforms the best on the ﬂuorescence task, whereas Evoformer is the best on the stability task.
Revised: In more detail, ESM-1b

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Revised: performs the best on the ﬂuorescence task, whereas Evoformer is the best on the stability task.

Original: This may be because protein stability has a closer relationship with protein structure, or becausesequence-only pretraining does not generalize as well as structural pretraining to de novo miniproteins.
Revised: This may be because protein stability has a closer relationship with protein structure, or becausesequence-only pretraining does not generalize as well as structural pretraining to de novo miniproteins.

Original: 4.3 Zero-shot Mutation effects Prediction
Revised: 4.3 Zero-shot Mutation effects Prediction

Original: Recent work revealed that PLMs are strong zero-shot learners in predicting potential viral mutations [19, 20].
Revised: Recent work revealed that PLMs are strong zero-shot learners in predicting potential viral mutations [19, 20].

Original: We are interested in evaluating Evoformer’s masked-residue reconstruction ability insuch a zero-shot setting given that it also has a BERT-like masked token prediction loss.
Revised: We are interested in evaluating Evoformer’s masked-residue reconstruction ability insuch a zero-shot setting given that it also has a BERT-like masked token prediction loss.

Original: Figure 4 shows the results of all PLMs.
Revised: Figure 4 shows the results of all PLMs.

Original: First, we observe that ESM-1b and ESM-1v [27] yield thebest results, and both of them are evolution-free models.
Revised: First, we observe that ESM-1b and ESM-1v [27] yield thebest results, and both of them are evolution-free models.

Original: ESM-1v shares an architecture with ESM-1b,but is an ensemble of 5 models trained on UniRef90 instead of UniRef50.
Revised: ESM-1v shares an architecture with ESM-1b,but is an ensemble of 5 models trained on UniRef90 instead of UniRef50.

Original: MSA-Transformer ingeneral performs well except on BLAT_ECOLX_Ostermeier2014.
Revised: MSA-Transformer ingeneral performs well except on BLAT_ECOLX_Ostermeier2014.

Original: We notice that MSA-Transformeris highly affected by the quality and sequence identity thresholds of MSAs.
Revised: We notice that MSA-Transformeris highly affected by the quality and sequence identity thresholds of MSAs.

Original: For our experiments, weuse the default setting by HHblits [33] searching from BFD 3 database (see Appendix A.3).
Revised: For our experiments, weuse the default setting by HHblits [33] searching from BFD 3 database (see Appendix A.3).

Original: The most surprising results are for Evoformer, which performs poorly in this task on all ﬁve datasets.
Revised: The most surprising results are for Evoformer, which performs poorly in this task on all ﬁve datasets.

Original: One of the key difference is that the amount of training protein data for Evoformer is 100 ⇥ smallerthan MSA-Transformer and ESM-1b (see Table 1).
Revised: One of the key difference is that the amount of training protein data for Evoformer is 100 ⇥ smallerthan MSA-Transformer and ESM-1b (see Table 1).

Original: To further validate this reason, we have presentresults of another two models, namely ESM1-85M (released by [35]) and ESM-PDB-88M (seemodel description in Figure 4) with a similar parameter size as AlphaFold.
Revised: To further validate this reason, we have presentresults of another two models, namely ESM1-85M (released by [35]) and ESM-PDB-88M (seemodel description in Figure 4) with a similar parameter size as AlphaFold.

Original: As shown, ESM1-85Mperforms comparably with ESM1b, while ESM-PDB-88M performs much worse, similar poor resultsas Evoformer.
Revised: As shown, ESM1-85Mperforms comparably with ESM1b, while ESM-PDB-88M performs much worse, similar poor resultsas Evoformer.

Original: This is because ESM-PDB-88M was trained with a much smaller dataset, i.e., thesame data as AlphaFold.
Revised: This is because ESM-PDB-88M was trained with a much smaller dataset, i.e., thesame data as AlphaFold.

Original: The could also explain the poor results of Evoformer as training on asmaller protein dataset is not enough to capture the mutation effects and evolution patterns.
Revised: The could also explain the poor results of Evoformer as training on asmaller protein dataset is not enough to capture the mutation effects and evolution patterns.

Original: Notethat there is also another key difference between these models.
Revised: Notethat there is also another key difference between these models.

Original: Evoformer or AlphaFold was trainedwith ﬁve loss functions, among which only one is the masked token reconstruction loss.
Revised: Evoformer or AlphaFold was trainedwith ﬁve loss functions, among which only one is the masked token reconstruction loss.

Original: However,
Revised: However,

Original: MSA-Transformer and ESM-1b are trained with only the the masked token reconstruction loss.
Revised: MSA-Transformer and ESM-1b are trained with only the the masked token reconstruction loss.

Original: Answer for Q(i):
Revised: Answer for Q(i):

Original: Yes .
Revised: Yes .

Original: The answer is provided in Table 4 and 5.
Revised: The answer is provided in Table 4 and 5.

Original: From these results, we can seethat Evoformer with pre-trained parameters largely outperforms the non-pretrained version in fourdifferent function prediction tasks.
Revised: From these results, we can seethat Evoformer with pre-trained parameters largely outperforms the non-pretrained version in fourdifferent function prediction tasks.

Original: Together with its structure representation ability, we concludethat parameters learned by AlphaFold are general-purpose and useful to various structure andfunction prediction tasks. 
Revised: Together with its structure representation ability, we concludethat parameters learned by AlphaFold are general-purpose and useful to various structure andfunction prediction tasks. https://bfd.mmseqs.

Original: [Table]
Revised: [Table]

Original: Answer for Q(ii): No .
Revised: Answer for Q(ii): No .

Original: The answer is provided in Table 3, 9, 4, 5 and Figure 3.
Revised: The answer is provided in Table 3, 9, 4, 5 and Figure 3.

Original: These resultsshow that Evoformer indeed outperforms MSA-Transformer and ESM-1b when predicting structures,but it does not obviously outperform ESM-1b when predicting function.
Revised: These resultsshow that Evoformer indeed outperforms MSA-Transformer and ESM-1b when predicting structures,but it does not obviously outperform ESM-1b when predicting function.

Original: In particular, it does notwork at all in the zero-shot mutation effect prediction task.
Revised: In particular, it does notwork at all in the zero-shot mutation effect prediction task.

Original: Accordingly, we can conclude thatthe AlphaFold-triggered revolution for structure prediction cannot be directly transferred tofunction predictions.
Revised: Accordingly, we can conclude thatthe AlphaFold-triggered revolution for structure prediction cannot be directly transferred tofunction predictions.

Original: ESM-1b is still the SOTA when predicting protein functions.
Revised: ESM-1b is still the SOTA when predicting protein functions.

Original: 4.4 Effect of MSA
Revised: 4.4 Effect of MSA

Original: Here, we want to understand the inﬂuence of MSAs on evolution-based PLMs.
Revised: Here, we want to understand the inﬂuence of MSAs on evolution-based PLMs.

Original: Further, we explore therelationship between evolution-based and -free models.
Revised: Further, we explore therelationship between evolution-based and -free models.

Original: As mentioned in Section 4.2, evolution-based
Revised: As mentioned in Section 4.2, evolution-based

Original: PLMs take only a single sequence as input for the supervised ﬁtness prediction tasks, so we do notconsider the affect of MSAs for these two tasks.
Revised: PLMs take only a single sequence as input for the supervised ﬁtness prediction tasks, so we do notconsider the affect of MSAs for these two tasks.

Original: We report results on Table 6 and Figure 5.
Revised: We report results on Table 6 and Figure 5.

Original: The MSA setting in Table 6 is consistent for both modeltraining and inference.
Revised: The MSA setting in Table 6 is consistent for both modeltraining and inference.

Original: For Figure 5, we only consider the inference stage because it is a zero-shottask.
Revised: For Figure 5, we only consider the inference stage because it is a zero-shottask.

Original: Clearly, we observe that without MSAs, Evoformer, and MSA-Transformer both yield muchworse results (with or without pre-training).
Revised: Clearly, we observe that without MSAs, Evoformer, and MSA-Transformer both yield muchworse results (with or without pre-training).

Original: Similarly, as shown in Figure 5, MSA-Transformer isalso affected by the number of MSAs.
Revised: Similarly, as shown in Figure 5, MSA-Transformer isalso affected by the number of MSAs.

Original: In general, it performs much worse with few MSAs.
Revised: In general, it performs much worse with few MSAs.

Original: Theseresults show that ( Answer for Q(iii) )
Revised: Theseresults show that ( Answer for Q(iii) )

Original: MSA data is a crucial input for evolution-based PLMs, fornot only the structure prediction but also the function prediction tasks.
Revised: MSA data is a crucial input for evolution-based PLMs, fornot only the structure prediction but also the function prediction tasks.

Original: 4.5 Relationship between Evolution-based & -free PLMs
Revised: 4.5 Relationship between Evolution-based & -free PLMs

Original: Evolution-based PLMs require MSA data as input for both training and inference.
Revised: Evolution-based PLMs require MSA data as input for both training and inference.

Original: However, databasesearch using HHblits [33] or Jackhmmer [22] can take up to 10 minutes for a 1000-residue protein.
Revised: However, databasesearch using HHblits [33] or Jackhmmer [22] can take up to 10 minutes for a 1000-residue protein.

Original: On the other hand, given the powerful function representation of ESM-1b, we design a fast homologyretrieval framework using Siamese ESM-1b, following [31].
Revised: On the other hand, given the powerful function representation of ESM-1b, we design a fast homologyretrieval framework using Siamese ESM-1b, following [31].

Original: Figure 6 is the schematic of the ESM-
Revised: Figure 6 is the schematic of the ESM-

Original: 1b-constructed MSA generating and serving process, called ESM-MSA (see Appendix A.2 fordetails).
Revised: 1b-constructed MSA generating and serving process, called ESM-MSA (see Appendix A.2 fordetails).

Original: [Table]
Revised: [Table]

Original: Table 7 shows the results by using different MSAs for Evoformer and MSA-Transformer.
Revised: Table 7 shows the results by using different MSAs for Evoformer and MSA-Transformer.

Original: MSAsgenerated by ESM-1b are comparable to those found by Jackhmmer (or HHblits) while being muchfaster (see Appendix A.3.2), suggesting that ( Answer for Q(iii) )
Revised: MSAsgenerated by ESM-1b are comparable to those found by Jackhmmer (or HHblits) while being muchfaster (see Appendix A.3.2), suggesting that ( Answer for Q(iii) )

Original: the function representation learnedby evolution-free ESM-1b can be used to construct MSAs for evolution-based MSA-Transformerand Evoformer.
Revised: the function representation learnedby evolution-free ESM-1b can be used to construct MSAs for evolution-based MSA-Transformerand Evoformer.

Original:  Conclusions and Limitations
Revised:  Conclusions and Limitations

Original: In the paper, we have presented empirical studies to characterize three successful PLMs.
Revised: In the paper, we have presented empirical studies to characterize three successful PLMs.

Original: In particular, we focus on the function representation capacity of Evoformer.
Revised: In particular, we focus on the function representation capacity of Evoformer.

Original: We draw several importantconclusions: (i) Evoformer encodes not only structure, but also various protein functional properties; it provides an alternative choice when predicting protein functions, especially the stabilityprediction task; (ii) despite that AlphaFold dominates the structure prediction tasks, Evoformer isnot yet able to substitute ESM-1b and MSA-Transformer when predicting protein functions; (iii) thefunctional predictability of evolution-based PLMs is also inﬂuenced by protein MSAs.
Revised: We draw several importantconclusions: (i) Evoformer encodes not only structure, but also various protein functional properties; it provides an alternative choice when predicting protein functions, especially the stabilityprediction task; (ii) despite that AlphaFold dominates the structure prediction tasks, Evoformer isnot yet able to substitute ESM-1b and MSA-Transformer when predicting protein functions; (iii) thefunctional predictability of evolution-based PLMs is also inﬂuenced by protein MSAs.

Original: Interestingly,evolution-free ESM-1b can generate sufﬁciently-accurate MSAs for evolution-based PLMs withmuch higher efﬁciency.
Revised: Interestingly,evolution-free ESM-1b can generate sufﬁciently-accurate MSAs for evolution-based PLMs withmuch higher efﬁciency.

Original: Our work points out both strengths and weakness of three SOTA PLMs,which are potentially useful to many downstream biological tasks, including but not limited to proteinengineering [8], cancer early detection [16], and drug discovery [24].
Revised: Our work points out both strengths and weakness of three SOTA PLMs,which are potentially useful to many downstream biological tasks, including but not limited to proteinengineering [8], cancer early detection [16], and drug discovery [24].

Original: An important limitation is that we probe the representation capacity of these PLMs by only adding alinear head on top — a common practice in the deep learning community.
Revised: An important limitation is that we probe the representation capacity of these PLMs by only adding alinear head on top — a common practice in the deep learning community.

Original: While this veriﬁes ourﬁndings, we believe such a linear layer is not sufﬁciently expressive to achieve SOTA results.
Revised: While this veriﬁes ourﬁndings, we believe such a linear layer is not sufﬁciently expressive to achieve SOTA results.

Original: That isthe reason why Evoformer with a eight-layer Structure module achieves much higher 3D structureaccuracy but with a linear layer it achieves less than 80% classiﬁcation accuracy for the secondarystructure prediction task.
Revised: That isthe reason why Evoformer with a eight-layer Structure module achieves much higher 3D structureaccuracy but with a linear layer it achieves less than 80% classiﬁcation accuracy for the secondarystructure prediction task.

Original: In addition, we do not fully disentangle the effects of the Evoformerarchitecture, training data, and loss function when comparing it to MSA-Transformer.
Revised: In addition, we do not fully disentangle the effects of the Evoformerarchitecture, training data, and loss function when comparing it to MSA-Transformer.

Original: Therefore, animportant direction is to develop more advanced functional modules for higher prediction accuracy.
Revised: Therefore, animportant direction is to develop more advanced functional modules for higher prediction accuracy.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?

Original: [Yes](b) Did you describe the limitations of your work?
Revised: [Yes](b) Did you describe the limitations of your work?

Original: [Yes](c)
Revised: [Yes](c)

Original: Did you discuss any potential negative societal impacts of your work?
Revised: Did you discuss any potential negative societal impacts of your work?

Original: [No](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: [No](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]2.
Revised: [Yes]2.

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A](b)
Revised: [N/A](b)

Original: Did you include complete proofs of all theoretical results?
Revised: Did you include complete proofs of all theoretical results?

Original: [N/A]3.
Revised: [N/A]3.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] We will releasecode and datasets for reproducibility.(b)
Revised: [Yes] We will releasecode and datasets for reproducibility.(b)

Original: Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
Revised: Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes] See Sections A.4 and A.5(c)
Revised: [Yes] See Sections A.4 and A.5(c)

Original: Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] In ﬁgure 5, we report the effect of different MSAs on theresults.(d)
Revised: [Yes] In ﬁgure 5, we report the effect of different MSAs on theresults.(d)

Original: Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] See appendix A.6.4.
Revised: [Yes] See appendix A.6.4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a)
Revised: (a)

Original: If your work uses existing assets, did you cite the creators
Revised: If your work uses existing assets, did you cite the creators?

Original: Yes](b
Revised: [Yes](b

Original: ) Did you mention the license of the assets?
Revised: ) Did you mention the license of the assets?

Original: [Yes
Revised: Yes

Original: ](c
Revised: (c

Original: ) Did you include any new assets either in the supplemental material or as a URL?
Revised: ) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]
Revised: [Yes]

Original: d) Did you discuss whether and 
Revised: d) Did you discuss whether and how

Original: how consent
Revised: consent

Original: was obtained from people whose data you’re
Revised: was obtained from people whose data you’re

Original: using /curating?
Revised: using/curating?

Original: [Yes]
Revised: [Yes]

Original: (e
Revised: (e

Original: ) Did you discuss whether 
Revised: ) Did you discuss whether the

Revised: data

Original: the data you are using/curating contains personally identiﬁableinformation or offensive content
Revised: you are using curating contains personally identiﬁable information or offensive content

Original: [N/A
Revised: [N/A

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...(

Original: (a
Revised: a)

Original: ) Did
Revised: Did

Original: you include the full text of instructions given to participants and screenshots, ifapplicable?
Revised: you include the full text of instructions given to participants and screenshots, ifapplicable?

Original: [N /A
Revised: [N/ A

Original: ](b ) Did you describe any
Revised: b) Did you describe any

Original: potential participant risks,
Revised: potential participant risks

Original:  with
Revised: , with

Original: links to Institutional Review
Revised: links to Institutional Review

Original: Board (IRB approvals,
Revised: Board (IRB approvals,

Original: if applicable?
Revised: if applicable?

Original: [N/A
Revised: [N/A

Original: c) Did you include the estimated hourly wage paid to participants and the total 
Revised: c) Did you include the estimated hourly wage paid to participants and the total amount

Original: amountspent
Revised: spent

Original: on participant compensation?
Revised: on participant compensation?

Original: [N/A
Revised: [N/A]