Original: Exploring evolution-based & -free protein language models as protein function predictors
Revised: Exploring evolution-based & -free protein language models as protein function predictors

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Large-scale Protein Language Models (PLMs) have improved performance inprotein prediction tasks, ranging from 3D structure prediction to various functionpredictions.
Revised: Large-scale Protein Language Models (PLMs) have improved performance inprotein prediction tasks, ranging from 3D structure prediction to various functionpredictions.

Original: In particular, AlphaFold [23], a ground-breaking AI system, couldpotentially reshape structural biology.
Revised: In particular, AlphaFold [20], a ground-breaking AI system, couldpotentially reshape structural biology.

Original: However, the utility of the PLM module in
Revised: However, the utility of the PLM module in

Original: AlphaFold, Evoformer, has not been explored beyond structure prediction.
Revised: AlphaFold, Evoformer, has not been explored beyond structure prediction.

Original: In thispaper, we investigate the representation ability of three popular PLMs: ESM-1b(single sequence) [35], MSA-Transformer (multiple sequence alignment) [30] and
Revised: In thispaper, we investigate the representation ability of three popular PLMs: ESM-1b(single sequence) [31], MSA-Transformer (multiple sequence alignment) [26] and

Original: Evoformer (structural), with a special focus on Evoformer.
Revised: Evoformer (structural), with a special focus on Evoformer.

Original: Speciﬁcally, we aimto answer the following key questions: (i) Does the Evoformer trained as partof AlphaFold produce representations amenable to predicting protein function? (ii)
Revised: Specifically, we aimto answer the following key questions: (i) Does the Evoformer trained as partof AlphaFold produce representations amenable to predicting protein function?(ii)

Original: If yes, can Evoformer replace ESM-1b and MSA-Transformer?
Revised: If yes, can Evoformer replace ESM-1b and MSA-Transformer?

Original: (iii)
Revised: (iii)

Original: How much do these PLMs rely on evolution-related protein data?
Revised: Howmuch do these PLMs rely on evolution-related protein data?

Original: In this regard, arethey complementary to each other?
Revised: In this regard, arethey complementary to each other?

Original: We compare these models by empirical studyalong with new insights and conclusions.
Revised: We compare these models by empirical studyalong with new insights and conclusions.

Original: Finally, we release code and datasets forreproducibility.
Revised: Finally, we release code and datasets forreproducibility.

Original:  Introduction
Revised:  Introduction

Original: Proteins perform the majority of biological activities.
Revised: Proteins perform the majority of biological activities.

Original: It is, therefore, crucial to decipher the mechanisms underlying their structural and functional properties.
Revised: It is, therefore, crucial to decipher the mechanisms underlying their structural and functional properties.

Original: The canonical sequence-structure-functionrelationship enables the success of sequence-based machine learning methods that infer protein structure and function from amino acid (AA) sequence.
Revised: The canonical sequence-structure-function relationship enables the success of sequence-based machine learning methods that infer protein structure and function from amino acid (AA) sequence.

Original: Large-scale protein language models (PLMs) withself-supervised pretraining on tens of millions to billions of proteins (PLMs) [28, 35, 13, 5] are thecurent state-of-the-art in predicting function and ﬁtness from sequences.
Revised: Large-scale protein language models (PLMs) withself-supervised pretraining on tens of millions to billions of proteins (PLMs) [24, 31, 11, 5] are thecurent state-of-the-art in predicting function and fitness from sequences.

Original: Meanwhile
Revised: Meanwhile

Original: AlphaFold [23], trained on experimental 3D protein structures from the Protein Data
Revised: AlphaFold [20], trained on experimental 3D protein structures from the Protein Data

Original: Bank (PDB) [39] can approach the resolution of experimental structures for most protein sequences.
Revised: Bank (PDB) [35] can approach the resolution of experimental structures for most protein sequences.

Original: Its multiple sequence alignment representation module, Evoformer, combines new deep learningmachinery, a PLM residue reconstruction task, and structural supervision in the form of a distogram.
Revised: Its multiple sequence alignment representation module, Evoformer, combines new deep learningmachinery, a PLM residue reconstruction task, and structural supervision in the form of a distogram.

Original: Like MSA-Transformer [30], Evoformer takes a family of evolutionarily-related and aligned proteinsequences as input, in contrast to PLMs such as ESM-1b [35] and TAPE [28], which only takeindividual protein sequences.
Revised: Like MSA-Transformer [26], Evoformer takes a family of evolutionarily-related and aligned proteinsequences as input, in contrast to PLMs such as ESM-1b [31] and TAPE [24], which only takeindividual protein sequences.

Original: Thus, for short, we refer to the former two models as evolution-based
Revised: Thus, for short, we refer to the former two models as evolution-based

Original: PLMs and the latter two as evolution-free PLMs.
Revised: PLMs and the latter two as evolution-free PLMs.

Original: Despite the remarkable success of AlphaFold in predicting structure from sequence, it is unknownwhether its Evoformer module can be applied to other problems, in particular predicting proteinfunction and ﬁtness.
Revised: Despite the remarkable success of AlphaFold in predicting structure from sequence, it is unknownwhether its Evoformer module can be applied to other problems, in particular predicting proteinfunction and fitness.

Original: Deciphering AlphaFold rather than treating it as a black box is beneﬁcial to both artiﬁcial intelligence and biology communities.
Revised: Deciphering AlphaFold rather than treating it as a black box is beneficial to both artificial intelligence and biology communities.

Original: Therefore, we attempt to answer the following keyquestions.
Revised: Therefore, we attempt to answer the following keyquestions.

Original: Q(i): Does the Evoformer trained during AlphaFold training learn general-purpose proteinrepresentations that can be used for various function prediction tasks?
Revised: Q(i): Does the Evoformer trained during AlphaFold training learn general-purpose proteinrepresentations that can be used for various function prediction tasks?

Original: Unlike ESM-1b and MSATransformer, Evoformer is trained with supervision from 3D structures.
Revised: Unlike ESM-1b and MSATransformer, Evoformer is trained with supervision from 3D structures.

Original: In addition, the second partof AlphaFold, the Structure Module, built on top of the 48 Evoformer blocks, is much more complexand deeper than the traditional (linear) classiﬁcation head used in ESM-1b and MSA-Transformer.
Revised: In addition, the second partof AlphaFold, the Structure Module, built on top of the 48 Evoformer blocks, is much more complexand deeper than the traditional (linear) classification head used in ESM-1b and MSA-Transformer.

Original: These differences make the function representation ability of Evoformer an open question.
Revised: These differences make the function representation ability of Evoformer an open question.

Original: Q(ii):
Revised: Q(ii):

Original: If Evoformer’s representation is general, does it outperform ESM-1b and MSATransformer on downstream tasks?
Revised: If Evoformer’s representation is general, does it outperform ESM-1b and MSATransformer on downstream tasks?

Original: While these three models are trained with different parametersizes and datasets (see Table 1), we believe the comparison results are still valuable because they arecurrently the most advanced PLMs.
Revised: While these three models are trained with different parametersizes and datasets (see Table 1), we believe the comparison results are still valuable because they arecurrently the most advanced PLMs.

Original: Training these large models from scratch is out of the reach ofmost academic research groups due to the compute and cost involved.
Revised: Training these large models from scratch is out of the reach ofmost academic research groups due to the compute and cost involved.

Original: In addition, we also investigatethe utility of ESM-1b and MSA-Transformer on function prediction tasks.
Revised: In addition, we also investigatethe utility of ESM-1b and MSA-Transformer on function prediction tasks.

Original: Q(iii): How much does the performance of evolution-based PLMs rely on the input MSAs?
Revised: Q(iii): How much does the performance of evolution-based PLMs rely on the input MSAs?

Original: Can evolution-free PLMs assist evolution-based PLMs in terms of MSA construction?
Revised: Can evolution-free PLMs assist evolution-based PLMs in terms of MSA construction?

Original: Both
Revised: Both

Original: Evoformer and MSA-Transformer take sets of aligned sequences (MSAs) as input.
Revised: Evoformer and MSA-Transformer take sets of aligned sequences (MSAs) as input.

Original: We investigatethe effect of MSA quality and depth on function prediction.
Revised: We investigatethe effect of MSA quality and depth on function prediction.

Original: On the other side, ESM-1b can simplyformulate MSA construction as the remote homology detection task.
Revised: On the other side, ESM-1b can simplyformulate MSA construction as the remote homology detection task.

Original: Then, it is interesting to knowwhether ESM1b-constructed MSAs can be used as inputs to Evoformer and MSA-Transformer.
Revised: Then, it is interesting to knowwhether ESM1b-constructed MSAs can be used as inputs to Evoformer and MSA-Transformer.

Original: We address the above questions through comprehensive empirical studies on a variety of structureand function prediction tasks.
Revised: We address the above questions through comprehensive empirical studies on a variety of structureand function prediction tasks.

Original: Note although AlphaFold (i.e. Evoformer + Structure Module) canaccurately predict protein 3D structures, we investigate the ability of Evoformer to perform otherstructure prediction tasks, such as secondary structure prediction and contact map prediction.
Revised: Note although AlphaFold (i.e. Evoformer + Structure Module) canaccurately predict protein 3D structures, we investigate the ability of Evoformer to perform otherstructure prediction tasks, such as secondary structure prediction and contact map prediction.

Original: Wemake the following key observations:
Revised: Wemake the following key observations:

Original: (i) The AlphaFold-trained Evoformer produces representations that are useful for both structure and function prediction, as shown on two structure prediction tasks, two function annotation tasks [48],and two ﬁtness score prediction tasks [28].(ii) Evoformer representations are useful for both protein-level and residue-level prediction tasks.
Revised: (i) The AlphaFold-trained Evoformer produces representations that are useful for both structure andfunction prediction, as shown on two structure prediction tasks, two function annotation tasks [42],and two fitness score prediction tasks [24]. (ii) Evoformer representations are useful for both protein-level and residue-level prediction tasks.(iii)

Original: (iii) Evoformer is superior to ESM-1b and MSA-Transformer for structure prediction and novel miniprotein stability prediction, but inferior to ESM-1b on other function prediction tasks.
Revised: Evoformer is superior to ESM-1b and MSA-Transformer for structure prediction and novelminiprotein stability prediction, but inferior to ESM-1b on other function prediction tasks.

Original: It performspoorly for zero-shot ﬁtness prediction tasks [27] compared to ESM-1b and MSA-Transformer.(iv)
Revised: It performspoorly for zero-shot fitness prediction tasks [23] compared to ESM-1b and MSA-Transformer.(iv)

Original: Evolution-based PLMs are superior to evolution-free ESM-1b model only in the structure prediction tasks, but in general, are worse than ESM-1b in most function prediction tasks anddatasets.
Revised: Evolution-based PLMs are superior to evolution-free ESM-1b model only in the structureprediction tasks, but in general, are worse than ESM-1b in most function prediction tasks anddatasets.

Original: (v) Consistent with structure prediction, evolution-based PLMs are also sensitive to the amount ofMSAs when predicting protein function.
Revised: (v) Consistent with structure prediction, evolution-based PLMs are also sensitive to the amount ofMSAs when predicting protein function.

Original: In addition, their performance using ESM-1b-constructedMSAs as input is comparable to the performance using MSAs generated by Jackhmmer [22] orHHblits [32].
Revised: In addition, their performance using ESM-1b-constructedMSAs as input is comparable to the performance using MSAs generated by Jackhmmer [19] orHHblits [28].

Original:  Related Work
Revised:  Related Work

Original: Protein language models The volume of protein data has exploded over the last decade with theadvancement of new DNA sequencing technologies.
Revised: Protein language models The volume of protein data has exploded over the last decade with theadvancement of new DNA sequencing technologies.

Original: Early work learned protein representationswith LSTMs [18, 4, 1].
Revised: Early work learned protein representationswith LSTMs [16, 4, 1].

Original: Recently, with the advent of large transformer models in natural languageprocessing (NLP), large PLMs using the Transformer [43] architecture and BERT [12] denoisingtask have been widely adopted [35, 28, 13].
Revised: Recently, with the advent of large transformer models in natural languageprocessing (NLP), large PLMs using the Transformer [38] architecture and BERT [10] denoisingtask have been widely adopted [31, 24, 11].

Original: 1 PLMs trained on large sequence databases have beensuccessfully applied for various protein related tasks, including secondary structure prediction [28,
Revised: 1 PLMs trained on large sequence databases have beensuccessfully applied for various protein related tasks, including secondary structure prediction [24,

Original: 18, 35, 13], contact prediction [29, 35], 3D structure prediction [23, 2], annotation prediction [7],signal peptide prediction [40], intracellular localization prediction [41], protein-protein interactionprediction [14]
Revised: 16, 31, 11], contact prediction [25, 31], 3D structure prediction [20, 2], annotation prediction [7],signal peptide prediction [36], intracellular localization prediction [37], protein-protein interactionprediction [12]

Original: [Table]
Revised: [Table]

Original: and ﬁtness prediction [1, 10, 19, 20, 27].
Revised: and fitness prediction [1, 9, 17, 18, 23].

Original: ESM-1b
Revised: ESM-1b

Original: [35] found that residue-residuecontacts can be recovered from the learned representation, identifying the close relationship between
Revised: [31] found that residue-residuecontacts can be recovered from the learned representation, identifying the close relationship between

Original: Transformer attention and biological features.
Revised: Transformer attention and biological features.

Original: Following this, [44] and [29] further studied theinterpretability of the attention map as contact map.
Revised: Following this, [39] and [25] further studied theinterpretability of the attention map as contact map.

Original: ProtTrans [13] benchmarked a variety of
Revised: ProtTrans [11] benchmarked a variety of

Original: BERT-like PLMs, including TAPE, ESM-1b,
Revised: BERT-like PLMs, including TAPE, ESM-1b,

Original: ProtTrans
Revised: ProtTrans

Original: [Table]
Revised: [Table]

Original: and MSA-Transformer.
Revised: and MSA-Transformer.

Original: Since ProtTrans withthe BERT architecture does not exceed ESM-1b, we choose to compare ESM-1b, MSA-Transformer,and Evoformer for this study.
Revised: Since ProtTrans withthe BERT architecture does not exceed ESM-1b, we choose to compare ESM-1b, MSA-Transformer,and Evoformer for this study.

Original: Table 1 and Figure 1 provide more details about the three models
Revised: Table 1 and Figure 1 provide more details about the three models

Original: [Table] [Table]
Revised: [Table] [Table]

Original: MSA
Revised: MSA

Original: Pair
Revised: Pair

Original: Evoformer (cid:11)(cid:23)(cid:27)(cid:3)(cid:69)(cid:79)(cid:82)(cid:70)(cid:78)(cid:86)(cid:12) Structure module (cid:11)(cid:27)(cid:3)(cid:69)(cid:79)(cid:82)(cid:70)(cid:78)(cid:86)(cid:12)
Revised: [Table]

Original: d. Alphafold
Revised: d. Alphafold

Original: Pair MSA
Revised: Pair MSA

Original: Seqence (cid:74)(cid:72)(cid:81)(cid:72)(cid:87)(cid:76)(cid:70)(cid:71)(cid:68)(cid:87)(cid:68)(cid:69)(cid:68)(cid:86)(cid:72)(cid:86)(cid:72)(cid:68)(cid:85)(cid:70)(cid:75) (cid:83)(cid:68)(cid:76)(cid:85)(cid:76)(cid:81)(cid:74)
Revised: [Table]

Original: Structure, function, and ﬁtness prediction A protein’s primary structure (i.e. AA sequence)determines its 3D structure, which in turn determines its functional properties.
Revised: Structure, function, and fitness prediction A protein’s primary structure (i.e. AA sequence)determines its 3D structure, which in turn determines its functional properties.

Original: This relationshipunderlies the success of PLMs, which infer protein structure and function from the raw sequence.
Revised: This relationshipunderlies the success of PLMs, which infer protein structure and function from the raw sequence.

Original: Secondary structure (SS) prediction [38] is an easier task than predicting the 2D contact map [45]and 3D structure [23].
Revised: Secondary structure (SS) prediction [34] is an easier task than predicting the 2D contact map [40]and 3D structure [20].

Original: MSA-Transformer and ESM-1b can make accurate unsupervised contactpredictions, which are an important input for the 3D structure generation.
Revised: MSA-Transformer and ESM-1b can make accurate unsupervised contactpredictions, which are an important input for the 3D structure generation.

Original: While remarkable progress has been made in structure prediction, it is unknown whether this
Revised: While remarkable progress has been made in structure prediction, it is unknown whether this

Original: AlphaFold-triggered revolution transfers to other tasks, in particular function prediction.
Revised: AlphaFold-triggered revolution transfers to other tasks, in particular function prediction.

Original: Proteinfunction is a broad term that refers to any biological or biochemical roles in organisms.
Revised: Proteinfunction is a broad term that refers to any biological or biochemical roles in organisms.

Original: In thispaper, we focus on functional annotation prediction (classiﬁcation task) [48, 6] and ﬁtness prediction(including both regression [10, 28] and zero-shot prediction [19, 27] tasks), as shown in Figure 2.
Revised: In thispaper, we focus on functional annotation prediction (classification task) [42, 6] and fitness prediction(including both regression [9, 24] and zero-shot prediction [17, 23] tasks), as shown in Figure 2.

Original: Compared with annotation prediction, ﬁtness prediction is more challenging since (1) there areroutinely few or no laboratory labels for supervision; (2) protein AA sequences are highly similargiven the same wild-type protein.
Revised: Compared with annotation prediction, fitness prediction is more challenging since (1) there areroutinely few or no laboratory labels for supervision; (2) protein AA sequences are highly similargiven the same wild-type protein.

Original:  Preliminaries
Revised:  Preliminaries

Original: [Figure]
Revised: [Figure]

Original: 3.1 Tasks and datasets
Revised: 3.1 Tasks and datasets

Original: [Figure]
Revised: [Figure]

Original: We evaluate models on two structure prediction tasks:
Revised: We evaluate models on two structure prediction tasks:

Original: Secondary structure (SS): This is a residue-level sequence-to-sequence task where each
Revised: Secondary structure (SS): This is a residue-level sequence-to-sequence task where each

Original: residue x i of a protein sequence x = { x 1 , x 2 , ..., x L } is mapped to a label y i correspondingto one of eight secondary structure tasks y i 2 { G, H, ..., C } [28].
Revised: residue x i of a protein sequence x = { x 1 , x 2 , ..., x L } is mapped to a label y i correspondingto one of eight secondary structure tasks y i ∈ { G, H, ..., C } [24].

Original: SS prediction examines
Revised: SS prediction examines

Original: the degree to which a PLM learns local structure.
Revised: the degree to which a PLM learns local structure.

Original: Contacts:
Revised: Contacts:

Original: For a given protein structure, two residues are considered to be in contact if their
Revised: For a given protein structure, two residues are considered to be in contact if their

Original: C  carbons are within 8Å. We evaluate on pairs that are more than 6 positions apart in the
Revised: C β carbons are within 8Å. We evaluate on pairs that are more than 6 positions apart in the

Original: primary structure [28].
Revised: primary structure [24].

Original: We measure the results using Precision@ L , which stands for the
Revised: We measure the results using Precision@ L , which stands for the

Original: precision for the top- L pairs with the highest predicted contact probability [47].
Revised: precision for the top- L pairs with the highest predicted contact probability [41].

Original: L is the
Revised: L is the

Original: length of the protein sequence.
Revised: length of the protein sequence.

Original: For both contacts and secondary structure, we use the dataset in [35] which is constructed from
Revised: For both contacts and secondary structure, we use the dataset in [31] which is constructed from

Original: SCOPe [15], and use the suggested split as the training and testing sets (see Table 2).
Revised: SCOPe [13], and use the suggested split as the training and testing sets (see Table 2).

Original: One concernis that the dataset used here has been trained by AlphaFold as they come from the Protein Data
Revised: One concernis that the dataset used here has been trained by AlphaFold as they come from the Protein Data

Original: Bank (PDB) [3].
Revised: Bank (PDB) [3].

Original: Hence, we investigate 48 additional proteins, which were collected from CAMEO 2(Continuous Automated Model EvaluatiOn) with ‘hard’ category from 2021-08-28 to 2022-04-30.
Revised: Hence, we investigate 48 additional proteins, which were collected from CAMEO 2(Continuous Automated Model EvaluatiOn) with ‘hard’ category from 2021-08-28 to 2022-04-30.

Original: We also evaluate on two function (annotation) classiﬁcation tasks:
Revised: We also evaluate on two function (annotation) classification tasks:

Original: Metal ion binding (MIB):
Revised: Metal ion binding (MIB):

Original: This is a binary classiﬁcation task, where a PLM with a newclassiﬁcation layer is used to determine whether there are metal ion–binding sites in theprotein.
Revised: This is a binary classification task, where a PLM with a newclassification layer is used to determine whether there are metal ion–binding sites in the protein.

Original: The dataset is also collected from PDB with annotation as metal ion binding.
Revised: The dataset is also collected from PDB with annotation as metal ion binding.

Original: Werandomly sample the same amount of proteins from the database as the negative class.
Revised: Werandomly sample the same amount of proteins from the database as the negative class.

Original: Antibiotic resistance (ABR):
Revised: Antibiotic resistance (ABR):

Original: This is a multi-class classiﬁcation task, where a PLM need tocorrectly determine which class of antibiotic a protein degrades.
Revised: This is a multi-class classification task, where a PLM need tocorrectly determine which class of antibiotic a protein degrades.

Original: We construct the datasetfrom CARD [ 26] which contains 19 different antibiotic types (see AppendixA.5 for details).
Revised: We construct the datasetfrom CARD [22] which contains 19 different antibiotic types (see Appendix for details).

Original: And three ﬁtness prediction tasks.
Revised: And three fitness prediction tasks.

Original: Unlike functional annotation prediction, protein sequences in thistask are all from the same wild-type with a small number of mutated residues.
Revised: Unlike functional annotation prediction, protein sequences in thistask are all from the same wild-type with a small number of mutated residues.

Original: Stability: This is a protein-level regression task that predicts the protease concentration atwhich a protein can maintain its fold [36].
Revised: Stability: This is a protein-level regression task that predicts the protease concentration at which a protein can maintain its fold [32].

Original: We use the data splits from TAPE.
Revised: We use the data splits from TAPE.

Original: Fluorescence: This is also a protein-level regression task, predicting the log-ﬂuorescence
Revised: Fluorescence: This is also a protein-level regression task, predicting the log-fluorescence

Original: intensity of the protein sequence [37].
Revised: intensity of the protein sequence [33].

Original: We use the data splits from TAPE.
Revised: We use the data splits from TAPE.

Original: Zero-shot mutation effect prediction: This is a protein-level prediction task by comparing the
Revised: Zero-shot mutation effect prediction: This is a protein-level prediction task by comparing the

Original: difference between likelihoods assigned to the mutated residue and the likelihoods assigned
Revised: difference between likelihoods assigned to the mutated residue and the likelihoods assigned

Original: to the wild-type (see [27] for details).
Revised: to the wild-type (see [23] for details).

Original: We evaluate ﬁve protein mutation datasets from
Revised: We evaluate five protein mutation datasets from

Original: DeepSequence [34]
Revised: DeepSequence [30].

Original: Only single point mutation data is considered in this sub-task
Revised: Only single point mutation data is considered in this sub-task

Original: 3.2 Methods
Revised: 3.2 Methods

Original: For supervised function prediction tasks, we remove the original classiﬁcation layer for ESM-1band MSA-Transformer, and the Structure module for AlphaFold.
Revised: For supervised function prediction tasks, we remove the original classification layer for ESM-1band MSA-Transformer, and the Structure module for AlphaFold.

Original: These remaining parts are calledprotein representations.
Revised: These remaining parts are calledprotein representations.

Original: We then add a linear layer on top of these representation models to performnew classiﬁcation or regression tasks.
Revised: We then add a linear layer on top of these representation models to performnew classification or regression tasks.

Original: We adopt the standard ﬁne-tuning strategy by ﬁne-tuning allparameters using Adamw optimizer with 1e-5 as learning rate.
Revised: We adopt the standard fine-tuning strategy by fine-tuning allparameters using Adamw optimizer with 1e-5 as learning rate.

Original: For the zero-shot ﬁtness predictiontask, no training is needed.
Revised: For the zero-shot fitness predictiontask, no training is needed.

Original: Instead, we obtain the softmax output at the corresponding mutation sitein each protein sequence.
Revised: Instead, we obtain the softmax output at the corresponding mutation sitein each protein sequence.

Original: The probability value is regarded as the ﬁtness value for each amino acid
Revised: The probability value is regarded as the fitness value for each amino acid

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Revised: Source

Revised: Train

Revised: Test

Original: [Table]
Revised: [Table]

Revised: SCOPe

Original: [Table]
Revised: [Table]

Revised: PDB

Revised: [Table]

Revised: CARD

Revised: [Table]

Original: TAPE
Revised: TAPE

Original: [Table]
Revised: [Table]

Original:   following [19, 27].
Revised: TAPE  following [17, 23].

Original: As for structure predictions, we follow the same practice in [28].
Revised: As for structure predictions, we follow the same practice in [24].

Original: Other detailscan be seen in Appendix A.1.
Revised: Other detailscan be seen in Appendix ?? .

Original:  Results
Revised:  Results

Original: 4.1 Structure Prediction
Revised: 4.1 Structure Prediction

Original: In this section, we examine the structure representation ability of the three PLMs models discussedabove.
Revised: In this section, we examine the structure representation ability of the three PLMs models discussedabove.

Original: In general, the secondary structure and contact prediction tasks do not have signiﬁcant practicalvalues since there is already highly accurate 3D structure data by AlphaFold.
Revised: In general, the secondary structure and contact prediction tasks do not have significant practicalvalues since there is already highly accurate 3D structure data by AlphaFold.

Original: The purpose instead isto provide a reference for function prediction, given that some tasks have similar formulation.
Revised: The purpose instead isto provide a reference for function prediction, given that some tasks have similar formulation.

Original: Table 3 and 9 (Appendix A.4) show the results of SS prediction and contact prediction, respectively.
Revised: Table 3 and ?? (Appendix ?? ) show the results of SS prediction and contact prediction, respectively.

Original: For SS prediction, we consider two settings: pre-trained parameters and training from scratch (i.e.,with random initialization).
Revised: For SS prediction, we consider two settings: pre-trained parameters and training from scratch (i.e.,with random initialization).

Original: For Evoformer, pre-training means the training of AlphaFold.
Revised: For Evoformer, pre-training means the training of AlphaFold.

Original: For contactprediction, all PLMs have been already pre-trained since there is no meaningful contact map withoutpre-training.
Revised: For contactprediction, all PLMs have been already pre-trained since there is no meaningful contact map withoutpre-training.

Original: [Table]
Revised: [Table]

Original: First, we can easily observe that with pre-training, Evoformer performs the best in both tasks.
Revised: First, we can easily observe that with pre-training, Evoformer performs the best in both tasks.

Original: Particularly, Evoformer outperforms ESM-1b and MSA-Transformer by a large margin for contactprediction with over 94% accuracy on Precision@ L (see Table 9).
Revised: Particularly, Evoformer outperforms ESM-1b and MSA-Transformer by a large margin for contactprediction with over 94% accuracy on Precision@ L (see Table ?? ).

Original: By contrast, it does not performas accurately as the contact prediction task for SS prediction, which improves ESM-1b by 11.6%and MSA-Transformer by 5%.
Revised: By contrast, it does not performas accurately as the contact prediction task for SS prediction, which improves ESM-1b by 11.6%and MSA-Transformer by 5%.

Original: This is reasonable since SS prediction is different from 3D structureprediction from the machine learning perspective.
Revised: This is reasonable since SS prediction is different from 3D structureprediction from the machine learning perspective.

Original: Even though Evoformer has strong structure representation ability, it may not work well by just adding a linear classiﬁcation layer.
Revised: Even though Evoformer has strong structure representation ability, it may not work well by just adding a linear classification layer.

Original: This suggests that amore complex structure module for SS prediction is necessary for higher accuracy.
Revised: This suggests that amore complex structure module for SS prediction is necessary for higher accuracy.

Original: In comparison,since contact information can be directly extracted from the pairwise distance map, it is not surprisingthat Evoformer shows superb results in this task.
Revised: In comparison,since contact information can be directly extracted from the pairwise distance map, it is not surprisingthat Evoformer shows superb results in this task.

Original: Thereby, we have also demonstrated results of 48new proteins in Figure 3, where these proteins have not been trained by AlphaFold.
Revised: Thereby, we have also demonstrated results of 48new proteins in Figure 3, where these proteins have not been trained by AlphaFold.

Original: In general, theyare consistent with our above analysis.
Revised: In general, theyare consistent with our above analysis.

Original: Second, we observe that pre-training is important: all PLMs are remarkably improved with around40% improvement for ESM-1b and 28% for Evoformer.
Revised: Second, we observe that pre-training is important: all PLMs are remarkably improved with around40% improvement for ESM-1b and 28% for Evoformer.

Original: Interestingly, we note that Evoformerperforms worse than MSA-Transformer by training from scratch.
Revised: Interestingly, we note that Evoformerperforms worse than MSA-Transformer by training from scratch.

Original: We conjecture that Evoformer ismuch harder to be trained without good initialization as its network architecture is more complex anddeeper than MSA-Transformer.
Revised: We conjecture that Evoformer ismuch harder to be trained without good initialization as its network architecture is more complex anddeeper than MSA-Transformer.

Original: Finally, for structure prediction, evolution-based models are largely better than evolution-free ESM-1b.
Revised: Finally, for structure prediction, evolution-based models are largely better than evolution-free ESM-1b.

Original: This is consistent with biological intuition and previous observation, such as in [30].
Revised: This is consistent with biological intuition and previous observation, such as in [26].

Original: 4.2 Supervised Function Prediction
Revised: 4.2 Supervised Function Prediction

Original: AlphaFold predicts very accurate protein structures.
Revised: AlphaFold predicts very accurate protein structures.

Original: However, it remains unknown whether itscore representation module Evoformer can prediction function.
Revised: However, it remains unknown whether itscore representation module Evoformer can prediction function.

Original: Similarly, we investigate MSATransformer’s ability to predict protein function, which is unknown either.
Revised: Similarly, we investigate MSATransformer’s ability to predict protein function, which is unknown either.

Original: Note that, unlike annotationprediction, evolution-based Evoformer and MSA-Transformer only take single sequences (rather than
Revised: Note that, unlike annotationprediction, evolution-based Evoformer and MSA-Transformer only take single sequences (rather than

Original: MSAs) as input since all protein variants share the same MSAs for the ﬁtness prediction task.
Revised: MSAs) as input since all protein variants share the same MSAs for the fitness prediction task.

Original: The results of annotation prediction (protein-level classiﬁcation) and ﬁtness predictions (protein-levelregression) are listed in Table 4 and 5 respectively.
Revised: The results of annotation prediction (protein-level classification) and fitness predictions (protein-levelregression) are listed in Table 4 and 5 respectively.

Original: We use classiﬁcation accuracy [28] for evaluatingannotation prediction and Spearman rank correlation [27] for evaluating ﬁtness prediction.
Revised: We use classification accuracy [24] for evaluatingannotation prediction and Spearman rank correlation [23] for evaluating fitness prediction.

Original: First, weﬁnd that for annotation prediction, ESM-1b, the evolution-free PLM, exceeds the two evolution-based
Revised: First, wefind that for annotation prediction, ESM-1b, the evolution-free PLM, exceeds the two evolution-based

Original: PLMs, although opposite observations are made for predicting structures.
Revised: PLMs, although opposite observations are made for predicting structures.

Original: Speciﬁcally, pre-trained
Revised: Specifically, pre-trained

Original: ESM-1b achieves 5.8% and 17.4% improvements over pre-trained Evoformer and MSA-Transformer.
Revised: ESM-1b achieves 5.8% and 17.4% improvements over pre-trained Evoformer and MSA-Transformer.

Original: This is a bit surprising since the biological intuition is that protein functional properties are mediatedby structures.
Revised: This is a bit surprising since the biological intuition is that protein functional properties are mediatedby structures.

Original: An important conclusion we reached here is that better structure PLMs do not meanthey have a better representation for predicting function.
Revised: An important conclusion we reached here is that better structure PLMs do not meanthey have a better representation for predicting function.

Original: Second, all pre-trained PLMs perform very well on the ABR task, a multi-class protein classiﬁcationtask.
Revised: Second, all pre-trained PLMs perform very well on the ABR task, a multi-class protein classificationtask.

Original: Despite that, we can still observe an obvious improvement between training-from-scratch andpre-training.
Revised: Despite that, we can still observe an obvious improvement between training-from-scratch andpre-training.

Original: Similar observation brought from pre-training can be obviously seen from all otherfunction prediction tasks.
Revised: Similar observation brought from pre-training can be obviously seen from all otherfunction prediction tasks.

Original: This suggests that both supervised and unsupervised pre-training on
Revised: This suggests that both supervised and unsupervised pre-training on

Original: PLMs are very useful to obtain protein function representations.
Revised: PLMs are very useful to obtain protein function representations.

Original: Third, we observe that the three pre-trained PLMs in general show better results than the one-hot [28]and ResNet [28, 17] baselines in Table 5.
Revised: Third, we observe that the three pre-trained PLMs in general show better results than the one-hot [24]and ResNet [24, 15] baselines in Table 5.

Original: This observation is new since neither Evoformer nor
Revised: This observation is new since neither Evoformer nor

Original: MSA-Transformer have been investigated for supervised ﬁtness prediction.
Revised: MSA-Transformer have been investigated for supervised fitness prediction.

Original: In more detail, ESM-1b
Revised: In more detail, ESM-1b

Original: [Table] [Table]
Revised: [Table] [Table]

Original: performs the best on the ﬂuorescence task, whereas Evoformer is the best on the stability task.
Revised: performs the best on the fluorescence task, whereas Evoformer is the best on the stability task.

Original: This may be because protein stability has a closer relationship with protein structure, or becausesequence-only pretraining does not generalize as well as structural pretraining to de novo miniproteins.
Revised: This may be because protein stability has a closer relationship with protein structure, or becausesequence-only pretraining does not generalize as well as structural pretraining to de novo miniproteins.

Original: 4.3 Zero-shot Mutation effects Prediction
Revised: 4.3 Zero-shot Mutation effects Prediction

Original: Recent work revealed that PLMs are strong zero-shot learners in predicting potential viral mutations [19, 20].
Revised: Recent work revealed that PLMs are strong zero-shot learners in predicting potential viral mutations [17, 18].

Original: We are interested in evaluating Evoformer’s masked-residue reconstruction ability insuch a zero-shot setting given that it also has a BERT-like masked token prediction loss.
Revised: We are interested in evaluating Evoformer’s masked-residue reconstruction ability insuch a zero-shot setting given that it also has a BERT-like masked token prediction loss.

Original: Figure 4 shows the results of all PLMs.
Revised: Figure 4 shows the results of all PLMs.

Original: First, we observe that ESM-1b and ESM-1v [27] yield thebest results, and both of them are evolution-free models.
Revised: First, we observe that ESM-1b and ESM-1v [23] yield thebest results, and both of them are evolution-free models.

Original: ESM-1v shares an architecture with ESM-1b,but is an ensemble of 5 models trained on UniRef90 instead of UniRef50.
Revised: ESM-1v shares an architecture with ESM-1b,but is an ensemble of 5 models trained on UniRef90 instead of UniRef50.

Original: MSA-Transformer ingeneral performs well except on BLAT_ECOLX_Ostermeier2014.
Revised: MSA-Transformer ingeneral performs well except on BLAT_ECOLX_Ostermeier2014.

Original: We notice that MSA-Transformeris highly affected by the quality and sequence identity thresholds of MSAs.
Revised: We notice that MSA-Transformeris highly affected by the quality and sequence identity thresholds of MSAs.

Original: For our experiments, weuse the default setting by HHblits [33] searching from BFD 3 database (see Appendix A.3).
Revised: For our experiments, weuse the default setting by HHblits [29] searching from BFD 3 database (see Appendix ?? ).

Original: The most surprising results are for Evoformer, which performs poorly in this task on all ﬁve datasets.
Revised: The most surprising results are for Evoformer, which performs poorly in this task on all five datasets.

Original: One of the key difference is that the amount of training protein data for Evoformer is 100 ⇥ smallerthan MSA-Transformer and ESM-1b (see Table 1).

Original: To further validate this reason, we have presentresults of another two models, namely ESM1-85M (released by [35]) and ESM-PDB-88M (seemodel description in Figure 4) with a similar parameter size as AlphaFold.
Revised: To further examine this cause, we present results of another two models, namely ESM1-85M (releasedby [31]) and ESM-PDB-88M (see model description in Figure 4) with a similar parameter sizeas AlphaFold and MSA-Transformer.

Original: As shown, ESM1-85Mperforms comparably with ESM1b, while ESM-PDB-88M performs much worse, similar poor resultsas Evoformer.
Revised: As shown, ESM1-85M performs comparably with ESM1band MSA-Transformer, which shows that the model size is not the reason for the poor results of

Original: This is because ESM-PDB-88M was trained with a much  smaller dataset, i.e., thesame data as AlphaFold. The could also explain the  poor resultsof Evoformer as training on asmaller protein dataset is not enough to capture the mutation effects and evolution patterns.
Revised: Evoformer. By contrast, ESM-PDB-88M performs much worse, indicating that smaller trainingdataset could substantially decrease the model performance. Given that AlphaFold was trained withnumerous protein sequences from Uniclust30 during the self-distillation process, we argue that datasize should not be a key reason for explaining its poor results as well.

Original: Notethat there is also another key difference between these models.
Revised: In fact, we notice there is alsoanother key difference between these models.

Original: Evoformer or AlphaFold was trainedwith ﬁve loss functions, among which  only one  is the masked token reconstruction  loss.
Revised: Evoformer or AlphaFold was trained with five lossfunctions, among which most are structure-related loss and only one auxiliary loss is the maskedtoken reconstruction (MTR) loss.

Original: However, MSA-Transformer and ESM-1b are trained with only the the masked token reconstruction loss.
Revised: By contrast, MSA-Transformer and ESM-1b were trained withonly the MTR loss.

Revised: This may explain the bad results of Evoformer for the zero-shot prediction tasksince it is highly correlated with the MTR loss.

Original: Answer for Q(i):
Revised: Answer for Q(i):

Original: Yes .
Revised: Yes .

Original: The answer is provided in Table 4 and 5.
Revised: The answer is provided in Table 4 and 5.

Original: From these results, we can seethat Evoformer with pre-trained parameters largely outperforms the non-pretrained version in fourdifferent function prediction tasks.
Revised: From these results, we can seethat Evoformer with pre-trained parameters largely outperforms the non-pretrained version in fourdifferent function prediction tasks.

Original: Together with its structure representation ability, we concludethat parameters learned by AlphaFold are general-purpose and useful to various structure andfunction prediction tasks. https://bfd.mmseqs.
Revised: Together with its structure representation ability, we conclude

Original: [Table]
Revised: [Table]

Revised: that parameters learned by AlphaFold are general-purpose and useful to various structure andfunction prediction tasks.

Original: Answer for Q(ii): No .
Revised: Answer for Q(ii): No .

Original: The answer is provided in Table 3, 9, 4, 5 and Figure 3.
Revised: The answer is provided in Table 3, ?? , 4, 5 and Figure 3.

Original: These resultsshow that Evoformer indeed outperforms MSA-Transformer and ESM-1b when predicting structures,but it does not obviously outperform ESM-1b when predicting function.
Revised: These resultsshow that Evoformer indeed outperforms MSA-Transformer and ESM-1b when predicting structures,but it does not obviously outperform ESM-1b when predicting function.

Original: In particular, it does notwork at all in the zero-shot mutation effect prediction task.
Revised: In particular, it does notwork at all in the zero-shot mutation effect prediction task.

Original: Accordingly, we can conclude thatthe AlphaFold-triggered revolution for structure prediction cannot be directly transferred tofunction predictions.
Revised: Accordingly, we can conclude thatthe AlphaFold-triggered revolution for structure prediction cannot be directly transferred tofunction predictions.

Original: ESM-1b is still the SOTA when predicting protein functions.
Revised: ESM-1b is still the SOTA when predicting protein functions.

Original: 4.4 Effect of MSA
Revised: 4.4 Effect of MSA

Original: Here, we want to understand the inﬂuence of MSAs on evolution-based PLMs.
Revised: Here, we want to understand the influence of MSAs on evolution-based PLMs.

Original: Further, we explore therelationship between evolution-based and -free models.
Revised: Further, we explore therelationship between evolution-based and -free models.

Original: As mentioned in Section 4.2, evolution-based
Revised: As mentioned in Section 4.2, evolution-based

Original: PLMs take only a single sequence as input for the supervised ﬁtness prediction tasks, so we do notconsider the affect of MSAs for these two tasks.
Revised: PLMs take only a single sequence as input for the supervised fitness prediction tasks, so we do notconsider the affect of MSAs for these two tasks.

Original: We report results on Table 6 and Figure 5.
Revised: We report results on Table 6 and Figure 5.

Original: The MSA setting in Table 6 is consistent for both modeltraining and inference.
Revised: The MSA setting in Table 6 is consistent for both modeltraining and inference.

Original: For Figure 5, we only consider the inference stage because it is a zero-shottask.
Revised: For Figure 5, we only consider the inference stage because it is a zero-shottask.

Original: Clearly, we observe that without MSAs, Evoformer, and MSA-Transformer both yield muchworse results (with or without pre-training).
Revised: Clearly, we observe that without MSAs, Evoformer, and MSA-Transformer both yield muchworse results (with or without pre-training).

Original: Similarly, as shown in Figure 5, MSA-Transformer isalso affected by the number of MSAs.
Revised: Similarly, as shown in Figure 5, MSA-Transformer isalso affected by the number of MSAs.

Original: In general, it performs much worse with few MSAs.
Revised: In general, it performs much worse with few MSAs.

Original: Theseresults show that ( Answer for Q(iii) )
Revised: Theseresults show that ( Answer for Q(iii) )

Original: MSA data is a crucial input for evolution-based PLMs, fornot only the structure prediction but also the function prediction tasks.
Revised: MSA data is a crucial input for evolution-based PLMs, fornot only the structure prediction but also the function prediction tasks.

Original: 4.5 Relationship between Evolution-based & -free PLMs
Revised: 4.5 Relationship between Evolution-based & -free PLMs

Original: Evolution-based PLMs require MSA data as input for both training and inference.
Revised: Evolution-based PLMs require MSA data as input for both training and inference.

Original: However, databasesearch using HHblits [33] or Jackhmmer [22] can take up to 10 minutes for a 1000-residue protein.
Revised: However, databasesearch using HHblits [29] or Jackhmmer [19] can take up to 10 minutes for a 1000-residue protein.

Original: On the other hand, given the powerful function representation of ESM-1b, we design a fast homologyretrieval framework using Siamese ESM-1b, following [31].
Revised: On the other hand, given the powerful function representation of ESM-1b, we design a fast homology retrieval framework using Siamese ESM-1b, following [27].

Original: Figure 6 is the schematic of theESM-
Revised: Figure ?? is the schematic of the

Original: 1b-constructed MSA generating and serving process, called ESM-MSA (see Appendix A.2 fordetails).
Revised: ESM-1b-constructed MSA generating and serving process, called ESM-MSA (see Appendix ?? ).

Original: [Table]
Revised: [Table]

Original: Table 7 shows the results by using different MSAs for Evoformer and MSA-Transformer.
Revised: Table 7 shows the results by using different MSAs for Evoformer and MSA-Transformer.

Original: MSAsgenerated by ESM-1b are comparable to those found by Jackhmmer (or HHblits) while being muchfaster (see Appendix A.3.2), suggesting that ( Answer for Q(iii) )
Revised: MSAsgenerated by ESM-1b are comparable to those found by Jackhmmer (or HHblits) while being muchfaster (see Appendix ?? ), suggesting that ( Answer for Q(iii) )

Original: the function representation learnedby evolution-free ESM-1b can be used to construct MSAs for evolution-based MSA-Transformerand Evoformer.
Revised: the function representation learned byevolution-free ESM-1b can be used to construct MSAs for evolution-based MSA-Transformerand Evoformer.

Original:  Conclusions and Limitations
Revised:  Conclusions and Limitations

Original: In the paper, we have presented empirical studies to characterize three successful PLMs.
Revised: In the paper, we have presented empirical studies to characterize three successful PLMs.

Original: In particular, we focus on the function representation capacity of Evoformer.
Revised: In particular, we focus on the function representation capacity of Evoformer.

Original: We draw several importantconclusions: (i) Evoformer encodes not only structure, but also various protein functional properties; it provides an alternative choice when predicting protein functions, especially the stabilityprediction task; (ii) despite that AlphaFold dominates the structure prediction tasks, Evoformer isnot yet able to substitute ESM-1b and MSA-Transformer when predicting protein functions; (iii) thefunctional predictability of evolution-based PLMs is also inﬂuenced by protein MSAs.
Revised: We draw several importantconclusions: (i) Evoformer encodes not only structure, but also various protein functional properties; it provides an alternative choice when predicting protein functions, especially the stabilityprediction task; (ii) despite that AlphaFold dominates the structure prediction tasks, Evoformer isnot yet able to substitute ESM-1b and MSA-Transformer when predicting protein functions; (iii) thefunctional predictability of evolution-based PLMs is also influenced by protein MSAs.

Original: Interestingly,evolution-free ESM-1b can generate sufﬁciently-accurate MSAs for evolution-based PLMs withmuch higher efﬁciency.
Revised: Interestingly,evolution-free ESM-1b can generate sufficiently-accurate MSAs for evolution-based PLMs withmuch higher efficiency.

Original: Our work points out both strengths and weakness of three SOTA PLMs,which are potentially useful to many downstream biological tasks, including but not limited to proteinengineering [8], cancer early detection [16], and drug discovery [24].
Revised: Our work points out both strengths and weakness of three SOTA PLMs,which are potentially useful to many downstream biological tasks, including but not limited to proteinengineering [8], cancer early detection [14], and drug discovery [21].

Original: An important limitation is that we probe the representation capacity of these PLMs by only adding alinear head on top — a common practice in the deep learning community.
Revised: An important limitation is that we probe the representation capacity of these PLMs by only adding alinear head on top — a common practice in the deep learning community.

Original: While this veriﬁes ourﬁndings, we believe such a linear layer is not sufﬁciently expressive to achieve SOTA results.
Revised: While this verifies ourfindings, we believe such a linear layer is not sufficiently expressive to achieve SOTA results.

Original: That isthe reason why Evoformer with a eight-layer Structure module achieves much higher 3D structureaccuracy but with a linear layer it achieves less than 80% classiﬁcation accuracy for the secondarystructure prediction task.
Revised: That isthe reason why Evoformer with a eight-layer Structure module achieves much higher 3D structureaccuracy but with a linear layer it achieves less than 80% classification accuracy for the secondarystructure prediction task.

Original: In addition, we do not fully disentangle the effects of the Evoformerarchitecture, training data, and loss function when comparing it to MSA-Transformer.
Revised: In addition, we do not fully disentangle the effects of the Evoformerarchitecture, training data, and loss function when comparing it to MSA-Transformer.

Original: Therefore, animportant direction is to develop more advanced functional modules for higher prediction accuracy.
Revised: Therefore, animportant direction is to develop more advanced functional modules for higher prediction accuracy.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?

Original: [Yes](b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes](c)
Revised: [Yes]

Original:  Did you discuss any potential negative societal impacts of your work? 
Revised: (c) Did you discuss any potential negative societal impacts of your work? [No]

Original: [No](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to [Table] them?

Original: [Yes]2.
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Revised: [Table]

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A](b)
Revised: [N/A]

Original:  Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]3.
Revised: [N/A]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?

Original: [Yes] We will releasecode and datasets for reproducibility.(b)
Revised: [Yes] We will release code and datasets for reproducibility.

Original:  Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes] See Sections A.4 and A.5(c)
Revised: [Yes] See Sections ?? and ??

Original:  Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?

Original: [Yes] In ﬁgure 5, we report the effect of different MSAs on theresults.(d)
Revised: [Yes] In figure 5, we report the effect of different MSAs on theresults.

Original:  Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] See appendix A.6.4.
Revised: [Yes] See appendix ?? .

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes](b
Revised: [Yes]

Original: ) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: Yes
Revised: [Yes]

Original: (c ) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]
Revised: [Yes]

Original: d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [Yes]
Revised: [Yes]

Original: (e

Original: ) Did you discuss whether the data you are using curating contains personally 
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: identiﬁable

Original: information or offensive content
Revised: information or offensive content?

Original: [N/A
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...(
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/ A
Revised: [N/A]

Original: b) Did you describe any potential participant risks
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: , with

Original: links to Institutional Review

Original: Board (IRB approvals,
Revised: Board (IRB) approvals,

Original: if applicable?

Original: [N/A

Original: c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent

Original: on participant compensation?

Original: [N/A]