Original: Exploring evolution-based & -free protein language models as protein function predictors
Revised: Exploring evolution-based & -free protein language models as protein function predictors

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Large-scale Protein Language Models (PLMs) have improved performance inprotein prediction tasks, ranging from 3D structure prediction to various functionpredictions.
Revised: Large-scale Protein Language Models (PLMs) have improved performance inprotein prediction tasks, ranging from 3D structure prediction to various functionpredictions.

Original: In particular, AlphaFold [20], a ground-breaking AI system, couldpotentially reshape structural biology.
Revised: In particular, AlphaFold [20], a ground-breaking AI system, couldpotentially reshape structural biology.

Original: However, the utility of the PLM module in
Revised: However, the utility of the PLM module in

Original: AlphaFold, Evoformer, has not been explored beyond structure prediction.
Revised: AlphaFold, Evoformer, has not been explored beyond structure prediction.

Original: In thispaper, we investigate the representation ability of three popular PLMs: ESM-1b(single sequence) [31], MSA-Transformer (multiple sequence alignment) [26] and
Revised: In thispaper, we investigate the representation ability of three popular PLMs: ESM-1b(single sequence) [31], MSA-Transformer (multiple sequence alignment) [26] and

Original: Evoformer (structural), with a special focus on Evoformer.
Revised: Evoformer (structural), with a special focus on Evoformer.

Original: Specifically, we aimto answer the following key questions: (i) Does the Evoformer trained as partof AlphaFold produce representations amenable to predicting protein function?(ii)
Revised: Specifically, we aimto answer the following key questions: (i) Does the Evoformer trained as partof AlphaFold produce representations amenable to predicting protein function?(ii)

Original: If yes, can Evoformer replace ESM-1b and MSA-Transformer?
Revised: If yes, can Evoformer replace ESM-1b and MSA-Transformer?

Original: (iii)
Revised: (iii)

Original: Howmuch do these PLMs rely on evolution-related protein data?
Revised: Howmuch do these PLMs rely on evolution-related protein data?

Original: In this regard, arethey complementary to each other?
Revised: In this regard, arethey complementary to each other?

Original: We compare these models by empirical studyalong with new insights and conclusions.
Revised: We compare these models by empirical studyalong with new insights and conclusions.

Original: Finally, we release code and datasets forreproducibility.
Revised: Finally, we release code and datasets forreproducibility.

Original:  Introduction
Revised:  Introduction

Original: Proteins perform the majority of biological activities.
Revised: Proteins perform the majority of biological activities.

Original: It is, therefore, crucial to decipher the mechanisms underlying their structural and functional properties.
Revised: It is, therefore, crucial to decipher the mechanisms underlying their structural and functional properties.

Original: The canonical sequence-structure-function relationship enables the success of sequence-based machine learning methods that infer protein structure and function from amino acid (AA) sequence.
Revised: The canonical sequence-structure-function relationship enables the success of sequence-based machine learning methods that infer protein structure and function from amino acid (AA) sequence.

Original: Large-scale protein language models (PLMs) withself-supervised pretraining on tens of millions to billions of proteins (PLMs) [24, 31, 11, 5] are thecurent state-of-the-art in predicting function and fitness from sequences.
Revised: Large-scale protein language models (PLMs) withself-supervised pretraining on tens of millions to billions of proteins (PLMs) [24, 31, 11, 5] are thecurent state-of-the-art in predicting function and fitness from sequences.

Original: Meanwhile
Revised: Meanwhile

Original: AlphaFold [20], trained on experimental 3D protein structures from the Protein Data
Revised: AlphaFold [20], trained on experimental 3D protein structures from the Protein Data

Original: Bank (PDB) [35] can approach the resolution of experimental structures for most protein sequences.
Revised: Bank (PDB) [35] can approach the resolution of experimental structures for most protein sequences.

Original: Its multiple sequence alignment representation module, Evoformer, combines new deep learningmachinery, a PLM residue reconstruction task, and structural supervision in the form of a distogram.
Revised: Its multiple sequence alignment representation module, Evoformer, combines new deep learningmachinery, a PLM residue reconstruction task, and structural supervision in the form of a distogram.

Original: Like MSA-Transformer [26], Evoformer takes a family of evolutionarily-related and aligned proteinsequences as input, in contrast to PLMs such as ESM-1b [31] and TAPE [24], which only takeindividual protein sequences.
Revised: Like MSA-Transformer [26], Evoformer takes a family of evolutionarily-related and aligned proteinsequences as input, in contrast to PLMs such as ESM-1b [31] and TAPE [24], which only takeindividual protein sequences.

Original: Thus, for short, we refer to the former two models as evolution-based
Revised: Thus, for short, we refer to the former two models as evolution-based

Original: PLMs and the latter two as evolution-free PLMs.
Revised: PLMs and the latter two as evolution-free PLMs.

Original: Despite the remarkable success of AlphaFold in predicting structure from sequence, it is unknownwhether its Evoformer module can be applied to other problems, in particular predicting proteinfunction and fitness.
Revised: Despite the remarkable success of AlphaFold in predicting structure from sequence, it is unknownwhether its Evoformer module can be applied to other problems, in particular predicting proteinfunction and fitness.

Original: Deciphering AlphaFold rather than treating it as a black box is beneficial to both artificial intelligence and biology communities.
Revised: Deciphering AlphaFold rather than treating it as a black box is beneficial to both artificial intelligence and biology communities.

Original: Therefore, we attempt to answer the following keyquestions.
Revised: Therefore, we attempt to answer the following keyquestions.

Original: Q(i): Does the Evoformer trained during AlphaFold training learn general-purpose proteinrepresentations that can be used for various function prediction tasks?
Revised: Q(i): Does the Evoformer trained during AlphaFold training learn general-purpose proteinrepresentations that can be used for various function prediction tasks?

Original: Unlike ESM-1b and MSATransformer, Evoformer is trained with supervision from 3D structures.
Revised: Unlike ESM-1b and MSATransformer, Evoformer is trained with supervision from 3D structures.

Original: In addition, the second partof AlphaFold, the Structure Module, built on top of the 48 Evoformer blocks, is much more complexand deeper than the traditional (linear) classification head used in ESM-1b and MSA-Transformer.
Revised: In addition, the second partof AlphaFold, the Structure Module, built on top of the 48 Evoformer blocks, is much more complexand deeper than the traditional (linear) classification head used in ESM-1b and MSA-Transformer.

Original: These differences make the function representation ability of Evoformer an open question.
Revised: These differences make the function representation ability of Evoformer an open question.

Original: Q(ii):
Revised: Q(ii):

Original: If Evoformer’s representation is general, does it outperform ESM-1b and MSATransformer on downstream tasks?
Revised: If Evoformer’s representation is general, does it outperform ESM-1b and MSATransformer on downstream tasks?

Original: While these three models are trained with different parametersizes and datasets (see Table 1), we believe the comparison results are still valuable because they arecurrently the most advanced PLMs.
Revised: While these three models are trained with different parametersizes and datasets (see Table 1), we believe the comparison results are still valuable because they arecurrently the most advanced PLMs.

Original: Training these large models from scratch is out of the reach ofmost academic research groups due to the compute and cost involved.
Revised: Training these large models from scratch is out of the reach ofmost academic research groups due to the compute and cost involved.

Original: In addition, we also investigatethe utility of ESM-1b and MSA-Transformer on function prediction tasks.
Revised: In addition, we also investigatethe utility of ESM-1b and MSA-Transformer on function prediction tasks.

Original: Q(iii): How much does the performance of evolution-based PLMs rely on the input MSAs?
Revised: Q(iii): How much does the performance of evolution-based PLMs rely on the input MSAs?

Original: Can evolution-free PLMs assist evolution-based PLMs in terms of MSA construction?
Revised: Can evolution-free PLMs assist evolution-based PLMs in terms of MSA construction?

Original: Both
Revised: Both

Original: Evoformer and MSA-Transformer take sets of aligned sequences (MSAs) as input.
Revised: Evoformer and MSA-Transformer take sets of aligned sequences (MSAs) as input.

Original: We investigatethe effect of MSA quality and depth on function prediction.
Revised: We investigatethe effect of MSA quality and depth on function prediction.

Original: On the other side, ESM-1b can simplyformulate MSA construction as the remote homology detection task.
Revised: On the other side, ESM-1b can simplyformulate MSA construction as the remote homology detection task.

Original: Then, it is interesting to knowwhether ESM1b-constructed MSAs can be used as inputs to Evoformer and MSA-Transformer.
Revised: Then, it is interesting to knowwhether ESM1b-constructed MSAs can be used as inputs to Evoformer and MSA-Transformer.

Original: We address the above questions through comprehensive empirical studies on a variety of structureand function prediction tasks.
Revised: We address the above questions through comprehensive empirical studies on a variety of structureand function prediction tasks.

Original: Note although AlphaFold (i.e. Evoformer + Structure Module) canaccurately predict protein 3D structures, we investigate the ability of Evoformer to perform otherstructure prediction tasks, such as secondary structure prediction and contact map prediction.
Revised: Note although AlphaFold (i.e. Evoformer + Structure Module) canaccurately predict protein 3D structures, we investigate the ability of Evoformer to perform otherstructure prediction tasks, such as secondary structure prediction and contact map prediction.

Original: Wemake the following key observations:
Revised: Wemake the following key observations:

Original: (i) The AlphaFold-trained Evoformer produces representations that are useful for both structure andfunction prediction, as shown on two structure prediction tasks, two function annotation tasks [42],and two fitness score prediction tasks [24].
Revised: (i) The AlphaFold-trained Evoformer produces representations that are useful for both structure andfunction prediction, as shown on two structure prediction tasks, two function annotation tasks [42],and two fitness score prediction tasks [24].

Original: (ii) Evoformer representations are useful for both protein-level and residue-level prediction tasks.(iii)
Revised: (ii) Evoformer representations are useful for both protein-level and residue-level prediction tasks.(iii)

Original: Evoformer is superior to ESM-1b and MSA-Transformer for structure prediction and novelminiprotein stability prediction, but inferior to ESM-1b on other function prediction tasks.
Revised: Evoformer is superior to ESM-1b and MSA-Transformer for structure prediction and novelminiprotein stability prediction, but inferior to ESM-1b on other function prediction tasks.

Original: It performspoorly for zero-shot fitness prediction tasks [23] compared to ESM-1b and MSA-Transformer.(iv)
Revised: It performspoorly for zero-shot fitness prediction tasks [23] compared to ESM-1b and MSA-Transformer.(iv)

Original: Evolution-based PLMs are superior to evolution-free ESM-1b model only in the structureprediction tasks, but in general, are worse than ESM-1b in most function prediction tasks anddatasets.
Revised: Evolution-based PLMs are superior to evolution-free ESM-1b model only in the structureprediction tasks, but in general, are worse than ESM-1b in most function prediction tasks anddatasets.

Original: (v) Consistent with structure prediction, evolution-based PLMs are also sensitive to the amount ofMSAs when predicting protein function.
Revised: (v) Consistent with structure prediction, evolution-based PLMs are also sensitive to the amount ofMSAs when predicting protein function.

Original: In addition, their performance using ESM-1b-constructedMSAs as input is comparable to the performance using MSAs generated by Jackhmmer [19] orHHblits [28].
Revised: In addition, their performance using ESM-1b-constructedMSAs as input is comparable to the performance using MSAs generated by Jackhmmer [19] orHHblits [28].

Original:  Related Work
Revised:  Related Work

Original: Protein language models The volume of protein data has exploded over the last decade with theadvancement of new DNA sequencing technologies.
Revised: Protein language models The volume of protein data has exploded over the last decade with theadvancement of new DNA sequencing technologies.

Original: Early work learned protein representationswith LSTMs [16, 4, 1].
Revised: Early work learned protein representationswith LSTMs [16, 4, 1].

Original: Recently, with the advent of large transformer models in natural languageprocessing (NLP), large PLMs using the Transformer [38] architecture and BERT [10] denoisingtask have been widely adopted [31, 24, 11].
Revised: Recently, with the advent of large transformer models in natural languageprocessing (NLP), large PLMs using the Transformer [38] architecture and BERT [10] denoisingtask have been widely adopted [31, 24, 11].

Original: 1 PLMs trained on large sequence databases have beensuccessfully applied for various protein related tasks, including secondary structure prediction [24,
Revised: 1 PLMs trained on large sequence databases have beensuccessfully applied for various protein related tasks, including secondary structure prediction [24,

Original: 16, 31, 11], contact prediction [25, 31], 3D structure prediction [20, 2], annotation prediction [7],signal peptide prediction [36], intracellular localization prediction [37], protein-protein interactionprediction [12]
Revised: 16, 31, 11], contact prediction [25, 31], 3D structure prediction [20, 2], annotation prediction [7],signal peptide prediction [36], intracellular localization prediction [37], protein-protein interactionprediction [12]

Original: [Table]
Revised: [Table]

Original: and fitness prediction [1, 9, 17, 18, 23].
Revised: and fitness prediction [1, 9, 17, 18, 23].

Original: ESM-1b
Revised: ESM-1b

Original: [31] found that residue-residuecontacts can be recovered from the learned representation, identifying the close relationship between
Revised: [31] found that residue-residuecontacts can be recovered from the learned representation, identifying the close relationship between

Original: Transformer attention and biological features.
Revised: Transformer attention and biological features.

Original: Following this, [39] and [25] further studied theinterpretability of the attention map as contact map.
Revised: Following this, [39] and [25] further studied theinterpretability of the attention map as contact map.

Original: ProtTrans [11] benchmarked a variety of
Revised: ProtTrans [11] benchmarked a variety of

Original: BERT-like PLMs, including TAPE, ESM-1b,
Revised: BERT-like PLMs, including TAPE, ESM-1b,

Original: ProtTrans
Revised: ProtTrans

Original: [Table]
Revised: [Table]

Original: and MSA-Transformer.
Revised: and MSA-Transformer.

Original: Since ProtTrans withthe BERT architecture does not exceed ESM-1b, we choose to compare ESM-1b, MSA-Transformer,and Evoformer for this study.
Revised: Since ProtTrans withthe BERT architecture does not exceed ESM-1b, we choose to compare ESM-1b, MSA-Transformer,and Evoformer for this study.

Original: Table 1 and Figure 1 provide more details about the three models
Revised: Table 1 and Figure 1 provide more details about the three models

Original: [Table] [Table]
Revised: [Table] [Table]

Original: MSA
Revised: MSA

Original: Pair
Revised: Pair

Original: [Table]
Revised: [Table]

Original: d. Alphafold
Revised: d. Alphafold

Original: Pair MSA
Revised: Pair MSA

Original: [Table]
Revised: [Table]

Original: Structure, function, and fitness prediction A protein’s primary structure (i.e. AA sequence)determines its 3D structure, which in turn determines its functional properties.
Revised: Structure, function, and fitness prediction A protein’s primary structure (i.e. AA sequence)determines its 3D structure, which in turn determines its functional properties.

Original: This relationshipunderlies the success of PLMs, which infer protein structure and function from the raw sequence.
Revised: This relationshipunderlies the success of PLMs, which infer protein structure and function from the raw sequence.

Original: Secondary structure (SS) prediction [34] is an easier task than predicting the 2D contact map [40]and 3D structure [20].
Revised: Secondary structure (SS) prediction [34] is an easier task than predicting the 2D contact map [40]and 3D structure [20].

Original: MSA-Transformer and ESM-1b can make accurate unsupervised contactpredictions, which are an important input for the 3D structure generation.
Revised: MSA-Transformer and ESM-1b can make accurate unsupervised contactpredictions, which are an important input for the 3D structure generation.

Original: While remarkable progress has been made in structure prediction, it is unknown whether this
Revised: While remarkable progress has been made in structure prediction, it is unknown whether this

Original: AlphaFold-triggered revolution transfers to other tasks, in particular function prediction.
Revised: AlphaFold-triggered revolution transfers to other tasks, in particular function prediction.

Original: Proteinfunction is a broad term that refers to any biological or biochemical roles in organisms.
Revised: Proteinfunction is a broad term that refers to any biological or biochemical roles in organisms.

Original: In thispaper, we focus on functional annotation prediction (classification task) [42, 6] and fitness prediction(including both regression [9, 24] and zero-shot prediction [17, 23] tasks), as shown in Figure 2.
Revised: In thispaper, we focus on functional annotation prediction (classification task) [42, 6] and fitness prediction(including both regression [9, 24] and zero-shot prediction [17, 23] tasks), as shown in Figure 2.

Original: Compared with annotation prediction, fitness prediction is more challenging since (1) there areroutinely few or no laboratory labels for supervision; (2) protein AA sequences are highly similargiven the same wild-type protein.
Revised: Compared with annotation prediction, fitness prediction is more challenging since (1) there areroutinely few or no laboratory labels for supervision; (2) protein AA sequences are highly similargiven the same wild-type protein.

Original:  Preliminaries
Revised:  Preliminaries

Original: [Figure]
Revised: [Figure]

Original: 3.1 Tasks and datasets
Revised: 3.1 Tasks and datasets

Original: [Figure]
Revised: [Figure]

Original: We evaluate models on two structure prediction tasks:
Revised: We evaluate models on two structure prediction tasks:

Original: Secondary structure (SS): This is a residue-level sequence-to-sequence task where each
Revised: Secondary structure (SS): This is a residue-level sequence-to-sequence task where each

Original: residue x i of a protein sequence x = { x 1 , x 2 , ..., x L } is mapped to a label y i correspondingto one of eight secondary structure tasks y i ∈ { G, H, ..., C } [24].
Revised: residue x i of a protein sequence x = { x 1 , x 2 , ..., x L } is mapped to a label y i correspondingto one of eight secondary structure tasks y i ∈ { G, H, ..., C } [24].

Original: SS prediction examines
Revised: SS prediction examines

Original: the degree to which a PLM learns local structure.
Revised: the degree to which a PLM learns local structure.

Original: Contacts:
Revised: Contacts:

Original: For a given protein structure, two residues are considered to be in contact if their
Revised: For a given protein structure, two residues are considered to be in contact if their

Original: C β carbons are within 8Å. We evaluate on pairs that are more than 6 positions apart in the
Revised: C β carbons are within 8Å. We evaluate on pairs that are more than 6 positions apart in the

Original: primary structure [24].
Revised: primary structure [24].

Original: We measure the results using Precision@ L , which stands for the
Revised: We measure the results using Precision@ L , which stands for the

Original: precision for the top- L pairs with the highest predicted contact probability [41].
Revised: precision for the top- L pairs with the highest predicted contact probability [41].

Original: L is the
Revised: L is the

Original: length of the protein sequence.
Revised: length of the protein sequence.

Original: For both contacts and secondary structure, we use the dataset in [31] which is constructed from
Revised: For both contacts and secondary structure, we use the dataset in [31] which is constructed from

Original: SCOPe [13], and use the suggested split as the training and testing sets (see Table 2).
Revised: SCOPe [13], and use the suggested split as the training and testing sets (see Table 2).

Original: One concernis that the dataset used here has been trained by AlphaFold as they come from the Protein Data
Revised: One concernis that the dataset used here has been trained by AlphaFold as they come from the Protein Data

Original: Bank (PDB) [3].
Revised: Bank (PDB) [3].

Original: Hence, we investigate 48 additional proteins, which were collected from CAMEO 2(Continuous Automated Model EvaluatiOn) with ‘hard’ category from 2021-08-28 to 2022-04-30.
Revised: Hence, we investigate 48 additional proteins, which were collected from CAMEO 2(Continuous Automated Model EvaluatiOn) with ‘hard’ category from 2021-08-28 to 2022-04-30.

Original: We also evaluate on two function (annotation) classification tasks:
Revised: We also evaluate on two function (annotation) classification tasks:

Original: Metal ion binding (MIB):
Revised: Metal ion binding (MIB):

Original: This is a binary classification task, where a PLM with a newclassification layer is used to determine whether there are metal ion–binding sites in the
Revised: This is a binary classification task, where a PLM with a newclassification layer is used to determine whether there are metal ion–binding sites in the

Original: protein.
Revised: protein.

Original: The dataset is also collected from PDB with annotation as metal ion binding.
Revised: The dataset is also collected from PDB with annotation as metal ion binding.

Original: Werandomly sample the same amount of proteins from the database as the negative class.
Revised: Werandomly sample the same amount of proteins from the database as the negative class.

Original: Antibiotic resistance (ABR):
Revised: Antibiotic resistance (ABR):

Original: This is a multi-class classification task, where a PLM need tocorrectly determine which class of antibiotic a protein degrades.
Revised: This is a multi-class classification task, where a PLM need tocorrectly determine which class of antibiotic a protein degrades.

Original: We construct the datasetfrom CARD [22] which contains 19 different antibiotic types (see Appendix
Revised: We construct the datasetfrom CARD [22] which contains 19 different antibiotic types (see Appendix

Original: for details).
Revised: for details).

Original: And three fitness prediction tasks.
Revised: And three fitness prediction tasks.

Original: Unlike functional annotation prediction, protein sequences in thistask are all from the same wild-type with a small number of mutated residues.
Revised: Unlike functional annotation prediction, protein sequences in thistask are all from the same wild-type with a small number of mutated residues.

Original: Stability: This is a protein-level regression task that predicts the protease concentration at
Revised: Stability: This is a protein-level regression task that predicts the protease concentration at

Original: which a protein can maintain its fold [32].
Revised: which a protein can maintain its fold [32].

Original: We use the data splits from TAPE.
Revised: We use the data splits from TAPE.

Original: Fluorescence: This is also a protein-level regression task, predicting the log-fluorescence
Revised: Fluorescence: This is also a protein-level regression task, predicting the log-fluorescence

Original: intensity of the protein sequence [33].
Revised: intensity of the protein sequence [33].

Original: We use the data splits from TAPE.
Revised: We use the data splits from TAPE.

Original: Zero-shot mutation effect prediction: This is a protein-level prediction task by comparing the
Revised: Zero-shot mutation effect prediction: This is a protein-level prediction task by comparing the

Original: difference between likelihoods assigned to the mutated residue and the likelihoods assigned
Revised: difference between likelihoods assigned to the mutated residue and the likelihoods assigned

Original: to the wild-type (see [23] for details).
Revised: to the wild-type (see [23] for details).

Original: We evaluate five protein mutation datasets from
Revised: We evaluate five protein mutation datasets from

Original: DeepSequence [30].
Revised: DeepSequence [30].

Original: Only single point mutation data is considered in this sub-task
Revised: Only single point mutation data is considered in this sub-task

Original: 3.2 Methods
Revised: 3.2 Methods

Original: For supervised function prediction tasks, we remove the original classification layer for ESM-1band MSA-Transformer, and the Structure module for AlphaFold.
Revised: For supervised function prediction tasks, we remove the original classification layer for ESM-1band MSA-Transformer, and the Structure module for AlphaFold.

Original: These remaining parts are calledprotein representations.
Revised: These remaining parts are calledprotein representations.

Original: We then add a linear layer on top of these representation models to performnew classification or regression tasks.
Revised: We then add a linear layer on top of these representation models to performnew classification or regression tasks.

Original: We adopt the standard fine-tuning strategy by fine-tuning allparameters using Adamw optimizer with 1e-5 as learning rate.
Revised: We adopt the standard fine-tuning strategy by fine-tuning allparameters using Adamw optimizer with 1e-5 as learning rate.

Original: For the zero-shot fitness predictiontask, no training is needed.
Revised: For the zero-shot fitness predictiontask, no training is needed.

Original: Instead, we obtain the softmax output at the corresponding mutation sitein each protein sequence.
Revised: Instead, we obtain the softmax output at the corresponding mutation sitein each protein sequence.

Original: The probability value is regarded as the fitness value for each amino acid
Revised: The probability value is regarded as the fitness value for each amino acid

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: Source
Revised: Source

Original: Train
Revised: Train

Original: Test
Revised: Test

Original: [Table]
Revised: [Table]

Original: SCOPe
Revised: SCOPe

Original: [Table]
Revised: [Table]

Original: PDB
Revised: PDB

Original: [Table]
Revised: [Table]

Original: CARD
Revised: CARD

Original: [Table]
Revised: [Table]

Original: TAPE
Revised: TAPE

Original: [Table]
Revised: [Table]

Original: TAPE  following [17, 23].
Revised: TAPE  following [17, 23].

Original: As for structure predictions, we follow the same practice in [24].
Revised: As for structure predictions, we follow the same practice in [24].

Original: Other detailscan be seen in Appendix ?? .
Revised: Other detailscan be seen in Appendix ?? .

Original:  Results
Revised:  Results

Original: 4.1 Structure Prediction
Revised: 4.1 Structure Prediction

Original: In this section, we examine the structure representation ability of the three PLMs models discussedabove.
Revised: In this section, we examine the structure representation ability of the three PLMs models discussedabove.

Original: In general, the secondary structure and contact prediction tasks do not have significant practicalvalues since there is already highly accurate 3D structure data by AlphaFold.
Revised: In general, the secondary structure and contact prediction tasks do not have significant practicalvalues since there is already highly accurate 3D structure data by AlphaFold.

Original: The purpose instead isto provide a reference for function prediction, given that some tasks have similar formulation.
Revised: The purpose instead isto provide a reference for function prediction, given that some tasks have similar formulation.

Original: Table 3 and ??
Revised: Table 3 and ??

Original: (Appendix ?? ) show the results of SS prediction and contact prediction, respectively.
Revised: (Appendix ?? ) show the results of SS prediction and contact prediction, respectively.

Original: For SS prediction, we consider two settings: pre-trained parameters and training from scratch (i.e.,with random initialization).
Revised: For SS prediction, we consider two settings: pre-trained parameters and training from scratch (i.e.,with random initialization).

Original: For Evoformer, pre-training means the training of AlphaFold.
Revised: For Evoformer, pre-training means the training of AlphaFold.

Original: For contactprediction, all PLMs have been already pre-trained since there is no meaningful contact map withoutpre-training.
Revised: For contactprediction, all PLMs have been already pre-trained since there is no meaningful contact map withoutpre-training.

Original: [Table]
Revised: [Table]

Original: First, we can easily observe that with pre-training, Evoformer performs the best in both tasks.
Revised: First, we can easily observe that with pre-training, Evoformer performs the best in both tasks.

Original: Particularly, Evoformer outperforms ESM-1b and MSA-Transformer by a large margin for contactprediction with over 94% accuracy on Precision@ L (see Table ?? ).
Revised: Particularly, Evoformer outperforms ESM-1b and MSA-Transformer by a large margin for contactprediction with over 94% accuracy on Precision@ L (see Table ?? ).

Original: By contrast, it does not performas accurately as the contact prediction task for SS prediction, which improves ESM-1b by 11.6%and MSA-Transformer by 5%.
Revised: By contrast, it does not performas accurately as the contact prediction task for SS prediction, which improves ESM-1b by 11.6%and MSA-Transformer by 5%.

Original: This is reasonable since SS prediction is different from 3D structureprediction from the machine learning perspective.
Revised: This is reasonable since SS prediction is different from 3D structureprediction from the machine learning perspective.

Original: Even though Evoformer has strong structure representation ability, it may not work well by just adding a linear classification layer.
Revised: Even though Evoformer has strong structure representation ability, it may not work well by just adding a linear classification layer.

Original: This suggests that amore complex structure module for SS prediction is necessary for higher accuracy.
Revised: This suggests that amore complex structure module for SS prediction is necessary for higher accuracy.

Original: In comparison,since contact information can be directly extracted from the pairwise distance map, it is not surprisingthat Evoformer shows superb results in this task.
Revised: In comparison,since contact information can be directly extracted from the pairwise distance map, it is not surprisingthat Evoformer shows superb results in this task.

Original: Thereby, we have also demonstrated results of 48new proteins in Figure 3, where these proteins have not been trained by AlphaFold.
Revised: Thereby, we have also demonstrated results of 48new proteins in Figure 3, where these proteins have not been trained by AlphaFold.

Original: In general, theyare consistent with our above analysis.
Revised: In general, theyare consistent with our above analysis.

Original: Second, we observe that pre-training is important: all PLMs are remarkably improved with around40% improvement for ESM-1b and 28% for Evoformer.
Revised: Second, we observe that pre-training is important: all PLMs are remarkably improved with around40% improvement for ESM-1b and 28% for Evoformer.

Original: Interestingly, we note that Evoformerperforms worse than MSA-Transformer by training from scratch.
Revised: Interestingly, we note that Evoformerperforms worse than MSA-Transformer by training from scratch.

Original: We conjecture that Evoformer ismuch harder to be trained without good initialization as its network architecture is more complex anddeeper than MSA-Transformer.
Revised: We conjecture that Evoformer ismuch harder to be trained without good initialization as its network architecture is more complex anddeeper than MSA-Transformer.

Original: Finally, for structure prediction, evolution-based models are largely better than evolution-free ESM-1b.
Revised: Finally, for structure prediction, evolution-based models are largely better than evolution-free ESM-1b.

Original: This is consistent with biological intuition and previous observation, such as in [26].
Revised: This is consistent with biological intuition and previous observation, such as in [26].

Original: 4.2 Supervised Function Prediction
Revised: 4.2 Supervised Function Prediction

Original: AlphaFold predicts very accurate protein structures.
Revised: AlphaFold predicts very accurate protein structures.

Original: However, it remains unknown whether itscore representation module Evoformer can prediction function.
Revised: However, it remains unknown whether itscore representation module Evoformer can prediction function.

Original: Similarly, we investigate MSATransformer’s ability to predict protein function, which is unknown either.
Revised: Similarly, we investigate MSATransformer’s ability to predict protein function, which is unknown either.

Original: Note that, unlike annotationprediction, evolution-based Evoformer and MSA-Transformer only take single sequences (rather than
Revised: Note that, unlike annotationprediction, evolution-based Evoformer and MSA-Transformer only take single sequences (rather than

Original: MSAs) as input since all protein variants share the same MSAs for the fitness prediction task.
Revised: MSAs) as input since all protein variants share the same MSAs for the fitness prediction task.

Original: The results of annotation prediction (protein-level classification) and fitness predictions (protein-levelregression) are listed in Table 4 and 5 respectively.
Revised: The results of annotation prediction (protein-level classification) and fitness predictions (protein-levelregression) are listed in Table 4 and 5 respectively.

Original: We use classification accuracy [24] for evaluatingannotation prediction and Spearman rank correlation [23] for evaluating fitness prediction.
Revised: We use classification accuracy [24] for evaluatingannotation prediction and Spearman rank correlation [23] for evaluating fitness prediction.

Original: First, wefind that for annotation prediction, ESM-1b, the evolution-free PLM, exceeds the two evolution-based
Revised: First, wefind that for annotation prediction, ESM-1b, the evolution-free PLM, exceeds the two evolution-based

Original: PLMs, although opposite observations are made for predicting structures.
Revised: PLMs, although opposite observations are made for predicting structures.

Original: Specifically, pre-trained
Revised: Specifically, pre-trained

Original: ESM-1b achieves 5.8% and 17.4% improvements over pre-trained Evoformer and MSA-Transformer.
Revised: ESM-1b achieves 5.8% and 17.4% improvements over pre-trained Evoformer and MSA-Transformer.

Original: This is a bit surprising since the biological intuition is that protein functional properties are mediatedby structures.
Revised: This is a bit surprising since the biological intuition is that protein functional properties are mediatedby structures.

Original: An important conclusion we reached here is that better structure PLMs do not meanthey have a better representation for predicting function.
Revised: An important conclusion we reached here is that better structure PLMs do not meanthey have a better representation for predicting function.

Original: Second, all pre-trained PLMs perform very well on the ABR task, a multi-class protein classificationtask.
Revised: Second, all pre-trained PLMs perform very well on the ABR task, a multi-class protein classificationtask.

Original: Despite that, we can still observe an obvious improvement between training-from-scratch andpre-training.
Revised: Despite that, we can still observe an obvious improvement between training-from-scratch andpre-training.

Original: Similar observation brought from pre-training can be obviously seen from all otherfunction prediction tasks.
Revised: Similar observation brought from pre-training can be obviously seen from all otherfunction prediction tasks.

Original: This suggests that both supervised and unsupervised pre-training on
Revised: This suggests that both supervised and unsupervised pre-training on

Original: PLMs are very useful to obtain protein function representations.
Revised: PLMs are very useful to obtain protein function representations.

Original: Third, we observe that the three pre-trained PLMs in general show better results than the one-hot [24]and ResNet [24, 15] baselines in Table 5.
Revised: Third, we observe that the three pre-trained PLMs in general show better results than the one-hot [24]and ResNet [24, 15] baselines in Table 5.

Original: This observation is new since neither Evoformer nor
Revised: This observation is new since neither Evoformer nor

Original: MSA-Transformer have been investigated for supervised fitness prediction.
Revised: MSA-Transformer have been investigated for supervised fitness prediction.

Original: In more detail, ESM-1b
Revised: In more detail, ESM-1b

Original: [Table] [Table]
Revised: [Table] [Table]

Original: performs the best on the fluorescence task, whereas Evoformer is the best on the stability task.
Revised: performs the best on the fluorescence task, whereas Evoformer is the best on the stability task.

Original: This may be because protein stability has a closer relationship with protein structure, or becausesequence-only pretraining does not generalize as well as structural pretraining to de novo miniproteins.
Revised: This may be because protein stability has a closer relationship with protein structure, or becausesequence-only pretraining does not generalize as well as structural pretraining to de novo miniproteins.

Original: 4.3 Zero-shot Mutation effects Prediction
Revised: 4.3 Zero-shot Mutation effects Prediction

Original: Recent work revealed that PLMs are strong zero-shot learners in predicting potential viral mutations [17, 18].
Revised: Recent work revealed that PLMs are strong zero-shot learners in predicting potential viral mutations [17, 18].

Original: We are interested in evaluating Evoformer’s masked-residue reconstruction ability insuch a zero-shot setting given that it also has a BERT-like masked token prediction loss.
Revised: We are interested in evaluating Evoformer’s masked-residue reconstruction ability insuch a zero-shot setting given that it also has a BERT-like masked token prediction loss.

Original: Figure 4 shows the results of all PLMs.
Revised: Figure 4 shows the results of all PLMs.

Original: First, we observe that ESM-1b and ESM-1v [23] yield thebest results, and both of them are evolution-free models.
Revised: First, we observe that ESM-1b and ESM-1v [23] yield thebest results, and both of them are evolution-free models.

Original: ESM-1v shares an architecture with ESM-1b,but is an ensemble of 5 models trained on UniRef90 instead of UniRef50.
Revised: ESM-1v shares an architecture with ESM-1b,but is an ensemble of 5 models trained on UniRef90 instead of UniRef50.

Original: MSA-Transformer ingeneral performs well except on BLAT_ECOLX_Ostermeier2014.
Revised: MSA-Transformer ingeneral performs well except on BLAT_ECOLX_Ostermeier2014.

Original: We notice that MSA-Transformeris highly affected by the quality and sequence identity thresholds of MSAs.
Revised: We notice that MSA-Transformeris highly affected by the quality and sequence identity thresholds of MSAs.

Original: For our experiments, weuse the default setting by HHblits [29] searching from BFD 3 database (see Appendix ?? ).
Revised: For our experiments, weuse the default setting by HHblits [29] searching from BFD 3 database (see Appendix ?? ).

Original: The most surprising results are for Evoformer, which performs poorly in this task on all five datasets.
Revised: The most surprising results are for Evoformer, which performs poorly in this task on all five datasets.

Original: Tofurther examine this cause, we present results of another two models, namely ESM1-85M (releasedby [31]) and ESM-PDB-88M (see model description in Figure 4) with a similar parameter sizeas AlphaFold and MSA-Transformer.
Revised: To examine this cause, we present results of another two models, namely ESM1-85M (releasedby [31]) and ESM-PDB-88M (see model description in Figure 4) with a similar parameter size as AlphaFold and MSA-Transformer.

Original: As shown, ESM1-85M performs comparably with ESM1band MSA-Transformer, which shows that the model size is not the reason for the poor results of Evoformer.
Revised: As shown, ESM1-85M performs comparably with ESM1b and MSA-Transformer, indicating that the model size is not the reason for the poor results of Evoformer.

Original: By contrast, ESM-PDB-88M performs much worse, indicating that smaller trainingdataset could substantially decrease the model performance.
Revised: By contrast, ESM-PDB-88M performs much worse, suggesting that smaller training dataset couldsubstantially decrease the model performance.

Original: Given that AlphaFold was trained withnumerous protein sequences from Uniclust30 during the self-distillation process, we argue that datasize should not be a key reason for explaining its poor results as well.
Revised: Given that AlphaFold was trained with numerousprotein sequences from Uniclust30 during the self-distillation process, we argue that data size shouldnot be a key reason for explaining its poor results either.

Original: In fact, we notice there is alsoanother key difference between these models.
Revised: Beyond these, there is another key differencebetween these models.

Original: Evoformer or AlphaFold was trained with five lossfunctions, among which most are structure-related loss and only one auxiliary loss is the maskedtoken reconstruction (MTR) loss.
Revised: Evoformer or AlphaFold was trained with five different loss functions, amongwhich four are structure-related loss and only one auxiliary loss is the masked token reconstruction(MTR) loss.

Original: By contrast, MSA-Transformer and ESM-1b were trained withonly the MTR loss.
Revised: By contrast, MSA-Transformer and ESM-1b were trained with only the MTR loss.

Original: This may explain the bad results of Evoformer for the zero-shot prediction tasksince it is highly correlated with the MTR loss.
Revised: Thismay explain the bad results of Evoformer for the mutational effect prediction task since the task ishighly correlated with the MTR loss.

Original: Answer for Q(i):
Revised: Answer for Q(i):

Original: Yes .
Revised: Yes .

Original: The answer is provided in Table 4 and 5.
Revised: The answer is provided in Table 4 and 5.

Original: From these results, we can seethat Evoformer with pre-trained parameters largely outperforms the non-pretrained version in fourdifferent function prediction tasks.
Revised: From these results, we can seethat Evoformer with pre-trained parameters largely outperforms the non-pretrained version in fourdifferent function prediction tasks.

Original: Together with its structure representation ability, we conclude
Revised: Together with its structure representation ability, we conclude

Original: [Table]
Revised: [Table]

Original: that parameters learned by AlphaFold are general-purpose and useful to various structure andfunction prediction tasks.
Revised: that parameters learned by AlphaFold are general-purpose and useful to various structure andfunction prediction tasks.

Original: Answer for Q(ii): No .
Revised: Answer for Q(ii): No .

Original: The answer is provided in Table 3, ??
Revised: The answer is provided in Table 3, ??

Original: , 4, 5 and Figure 3.
Revised: , 4, 5 and Figure 3.

Original: These resultsshow that Evoformer indeed outperforms MSA-Transformer and ESM-1b when predicting structures,but it does not obviously outperform ESM-1b when predicting function.
Revised: These resultsshow that Evoformer indeed outperforms MSA-Transformer and ESM-1b when predicting structures,but it does not obviously outperform ESM-1b when predicting function.

Original: In particular, it does notwork at all in the zero-shot mutation effect prediction task.
Revised: In particular, it does notwork at all in the zero-shot mutation effect prediction task.

Original: Accordingly, we can conclude thatthe AlphaFold-triggered revolution for structure prediction cannot be directly transferred tofunction predictions.
Revised: Accordingly, we can conclude thatthe AlphaFold-triggered revolution for structure prediction cannot be directly transferred tofunction predictions.

Original: ESM-1b is still the SOTA when predicting protein functions.
Revised: ESM-1b is still the SOTA when predicting protein functions.

Original: 4.4 Effect of MSA
Revised: 4.4 Effect of MSA

Original: Here, we want to understand the influence of MSAs on evolution-based PLMs.
Revised: Here, we want to understand the influence of MSAs on evolution-based PLMs.

Original: Further, we explore therelationship between evolution-based and -free models.
Revised: Further, we explore therelationship between evolution-based and -free models.

Original: As mentioned in Section 4.2, evolution-based
Revised: As mentioned in Section 4.2, evolution-based

Original: PLMs take only a single sequence as input for the supervised fitness prediction tasks, so we do notconsider the affect of MSAs for these two tasks.
Revised: PLMs take only a single sequence as input for the supervised fitness prediction tasks, so we do notconsider the affect of MSAs for these two tasks.

Original: We report results on Table 6 and Figure 5.
Revised: We report results on Table 6 and Figure 5.

Original: The MSA setting in Table 6 is consistent for both modeltraining and inference.
Revised: The MSA setting in Table 6 is consistent for both modeltraining and inference.

Original: For Figure 5, we only consider the inference stage because it is a zero-shottask.
Revised: For Figure 5, we only consider the inference stage because it is a zero-shottask.

Original: Clearly, we observe that without MSAs, Evoformer, and MSA-Transformer both yield muchworse results (with or without pre-training).
Revised: Clearly, we observe that without MSAs, Evoformer, and MSA-Transformer both yield muchworse results (with or without pre-training).

Original: Similarly, as shown in Figure 5, MSA-Transformer isalso affected by the number of MSAs.
Revised: Similarly, as shown in Figure 5, MSA-Transformer isalso affected by the number of MSAs.

Original: In general, it performs much worse with few MSAs.
Revised: In general, it performs much worse with few MSAs.

Original: Theseresults show that ( Answer for Q(iii) )
Revised: Theseresults show that ( Answer for Q(iii) )

Original: MSA data is a crucial input for evolution-based PLMs, fornot only the structure prediction but also the function prediction tasks.
Revised: MSA data is a crucial input for evolution-based PLMs, fornot only the structure prediction but also the function prediction tasks.

Original: 4.5 Relationship between Evolution-based & -free PLMs
Revised: 4.5 Relationship between Evolution-based & -free PLMs

Original: Evolution-based PLMs require MSA data as input for both training and inference.
Revised: Evolution-based PLMs require MSA data as input for both training and inference.

Original: However, databasesearch using HHblits [29] or Jackhmmer [19] can take up to 10 minutes for a 1000-residue protein.
Revised: However, databasesearch using HHblits [29] or Jackhmmer [19] can take up to 10 minutes for a 1000-residue protein.

Original: On the other hand, given the powerful function representation of ESM-1b, we design a fast homology retrieval framework using Siamese ESM-1b, following [27].
Revised: On the other hand, given the powerful function representation of ESM-1b, we design a fast homology retrieval framework using Siamese ESM-1b, following [27].

Original: Figure ??
Revised: Figure ??

Original: is the schematic of the
Revised: is the schematic of the

Original: ESM-1b-constructed MSA generating and serving process, called ESM-MSA (see Appendix ?? ).
Revised: ESM-1b-constructed MSA generating and serving process, called ESM-MSA (see Appendix ?? ).

Original: [Table]
Revised: [Table]

Original: Table 7 shows the results by using different MSAs for Evoformer and MSA-Transformer.
Revised: Table 7 shows the results by using different MSAs for Evoformer and MSA-Transformer.

Original: MSAsgenerated by ESM-1b are comparable to those found by Jackhmmer (or HHblits) while being muchfaster (see Appendix ?? ), suggesting that ( Answer for Q(iii) )
Revised: MSAsgenerated by ESM-1b are comparable to those found by Jackhmmer (or HHblits) while being muchfaster (see Appendix ?? ), suggesting that ( Answer for Q(iii) )

Original: the function representation learned byevolution-free ESM-1b can be used to construct MSAs for evolution-based MSA-Transformerand Evoformer.
Revised: the function representation learned byevolution-free ESM-1b can be used to construct MSAs for evolution-based MSA-Transformerand Evoformer.

Original:  Conclusions and Limitations
Revised:  Conclusions and Limitations

Original: In the paper, we have presented empirical studies to characterize three successful PLMs.
Revised: In the paper, we have presented empirical studies to characterize three successful PLMs.

Original: In particular, we focus on the function representation capacity of Evoformer.
Revised: In particular, we focus on the function representation capacity of Evoformer.

Original: We draw several importantconclusions: (i) Evoformer encodes not only structure, but also various protein functional properties; it provides an alternative choice when predicting protein functions, especially the stabilityprediction task; (ii) despite that AlphaFold dominates the structure prediction tasks, Evoformer isnot yet able to substitute ESM-1b and MSA-Transformer when predicting protein functions; (iii) thefunctional predictability of evolution-based PLMs is also influenced by protein MSAs.
Revised: We draw several importantconclusions: (i) Evoformer encodes not only structure, but also various protein functional properties; it provides an alternative choice when predicting protein functions, especially the stabilityprediction task; (ii) despite that AlphaFold dominates the structure prediction tasks, Evoformer isnot yet able to substitute ESM-1b and MSA-Transformer when predicting protein functions; (iii) thefunctional predictability of evolution-based PLMs is also influenced by protein MSAs.

Original: Interestingly,evolution-free ESM-1b can generate sufficiently-accurate MSAs for evolution-based PLMs withmuch higher efficiency.
Revised: Interestingly,evolution-free ESM-1b can generate sufficiently-accurate MSAs for evolution-based PLMs withmuch higher efficiency.

Original: Our work points out both strengths and weakness of three SOTA PLMs,which are potentially useful to many downstream biological tasks, including but not limited to proteinengineering [8], cancer early detection [14], and drug discovery [21].
Revised: Our work points out both strengths and weakness of three SOTA PLMs,which are potentially useful to many downstream biological tasks, including but not limited to proteinengineering [8], cancer early detection [14], and drug discovery [21].

Original: An important limitation is that we probe the representation capacity of these PLMs by only adding alinear head on top — a common practice in the deep learning community.
Revised: An important limitation is that we probe the representation capacity of these PLMs by only adding alinear head on top — a common practice in the deep learning community.

Original: While this verifies ourfindings, we believe such a linear layer is not sufficiently expressive to achieve SOTA results.
Revised: While this verifies ourfindings, we believe such a linear layer is not sufficiently expressive to achieve SOTA results.

Original: That isthe reason why Evoformer with a eight-layer Structure module achieves much higher 3D structureaccuracy but with a linear layer it achieves less than 80% classification accuracy for the secondarystructure prediction task.
Revised: That isthe reason why Evoformer with a eight-layer Structure module achieves much higher 3D structureaccuracy but with a linear layer it achieves less than 80% classification accuracy for the secondarystructure prediction task.

Original: In addition, we do not fully disentangle the effects of the Evoformerarchitecture, training data, and loss function when comparing it to MSA-Transformer.
Revised: In addition, we do not fully disentangle the effects of the Evoformerarchitecture, training data, and loss function when comparing it to MSA-Transformer.

Original: Therefore, animportant direction is to develop more advanced functional modules for higher prediction accuracy.
Revised: Therefore, animportant direction is to develop more advanced functional modules for higher prediction accuracy.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [No]
Revised: [No]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: [Table]
Revised: [Table]

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: [Table]
Revised: [Table]

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes] We will release
Revised: [Yes] We will release

Original: code and datasets for reproducibility.
Revised: code and datasets for reproducibility.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes] See Sections ??
Revised: [Yes] See Sections ??

Original: and ??
Revised: and ??

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi-
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experi-

Original: ments multiple times)?
Revised: ments multiple times)?

Original: [Yes] In figure 5, we report the effect of different MSAs on theresults.
Revised: [Yes] In figure 5, we report the effect of different MSAs on theresults.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] See appendix ?? .
Revised: [Yes] See appendix ?? .

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]
Revised: [Yes]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [Yes]
Revised: [Yes]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals,
Revised: Board (IRB) approvals,