Original: A single-cell gene expression language model
Revised: A single-cell gene expression language model

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Gene regulation is a dynamic process that connects genotype and phenotype.
Revised: Gene regulation is a dynamic process that connects genotype and phenotype.

Original: Giventhe difficulty of physically mapping mammalian gene circuitry , we require newcomputational methods to learn regulatory rules.
Revised: Given the difficulty of physically mapping mammalian gene circuitry, we require new computational methods to learn regulatory rules.

Original: Natural language is a valuableanalogy to the communication of regulatory control.
Revised: Natural language is a valuable analogy to the communication of regulatory control.

Original: Machine learning systemsmodel natural language by explicitly learning context dependencies between words.
Revised: Machine learning systems model natural language by explicitly learning context dependencies between words.

Original: We propose a similar system applied to single-cell RNA expression profiles to learncontext dependencies between genes.
Revised: We propose a similar system applied to single-cell RNA expression profiles to learn context dependencies between genes.

Original: Our model, Exceiver, is trained across adiversity of cell types using a self-supervised task formulated for discrete count data,accounting for feature sparsity.
Revised: Our model, Exceiver, is trained across a diversity of cell types using a self-supervised task formulated for discrete count data, accounting for feature sparsity.

Original: We found agreement between the similarity profilesof latent sample representations and learned gene embeddings with respect tobiological annotations.
Revised: We found agreement between the similarity profiles of latent sample representations and learned gene embeddings with respect to biological annotations.

Original: We evaluated Exceiver on a new dataset and a downstreamprediction task and found that pretraining supports transfer learning.
Revised: We evaluated Exceiver on a new dataset and a downstream prediction task and found that pretraining supports transfer learning.

Original: Our workprovides a framework to model gene regulation on a single-cell level and transferknowledge to downstream tasks.
Revised: Our work provides a framework to model gene regulation on a single-cell level and transfer knowledge to downstream tasks.

Original:  Introduction
Revised:  Introduction

Original: Many biological processes regulate the relationship between genotype and phenotype.
Revised: Many biological processes regulate the relationship between genotype and phenotype.

Original: On onehand, classical genetics defines simple hereditary rules.
Revised: On one hand, classical genetics defines simple hereditary rules.

Original: On the other hand, complex regulatorynetworks mediate response to the environment.
Revised: On the other hand, complex regulatory networks mediate response to the environment.

Original: Eventually, we may hope to model molecular circuitrycomprehensively to accurately predict phenotypes.
Revised: Eventually, we may hope to model molecular circuitry comprehensively to accurately predict phenotypes.

Original: In this direction, learned generalizations ofbiological processes may support medical interventions such as individual disease risk prediction andpatient therapy selection.
Revised: In this direction, learned generalizations of biological processes may support medical interventions such as individual disease risk prediction and patient therapy selection.

Original: Large-scale cellular assays capture snapshots of complex and dynamic biological processes such asgene regulation, with RNA abundances being one measurable outcome.
Revised: Large-scale cellular assays capture snapshots of complex and dynamic biological processes such as gene regulation, with RNA abundances being one measurable outcome.

Original: Single-cell RNA sequencing(scRNA-seq) observations can relate cellular states and mRNA expression relationships, revealinggene programs corresponding to disease processes, genetic perturbations, and therapeutic interventions [4].
Revised: Single-cell RNA sequencing (scRNA-seq) observations can relate cellular states and mRNA expression relationships, revealing gene programs corresponding to disease processes, genetic perturbations and therapeutic interventions [4].

Original: Given the difficulty of physically mapping regulatory circuitry explicitly, we hypothesizeda model trained on a large volume of transcriptomic profiles would instead implicitly learn RNAexpression dependencies that reflect regulatory logic.
Revised: Given the difficulty of physically mapping regulatory circuitry explicitly, we hypothesized a model trained on a large volume of transcriptomic profiles would instead implicitly learn RNA expression dependencies that reflect regulatory logic.

Original: Pretrained models in natural language processing, computer vision, and protein modeling motivate asimilar approach in systems biology [2, 3, 8].
Revised: Pretrained models in natural language processing, computer vision, and protein modeling motivate a similar approach in systems biology [2, 3, 8].

Original: Pretrained models that transfer to downstream tasksshare three components leveraging domain-specific inductive biases.
Revised: Pretrained models that transfer to downstream tasks share three components leveraging domain-specific inductive biases.

Original: First, sufficient unlabeled datavolumes provide enough information for highly parameterized models to learn complex relationshipsbetween features.
Revised: First, sufficient unlabeled data volumes provide enough information for highly parameterized models to learn complex relationships between features.

Original: Second, models learn from unlabeled data in a self-supervised manner, often byfeature masking, in which unmasked features are used to predict a fraction of values that are masked.
Revised: Second, models learn from unlabeled data in a self-supervised manner, often by feature masking, in which unmasked features are used to predict a fraction of values that are masked.

Original: Third, an attention mechanism learns the dependencies between features.
Revised: Third, an attention mechanism learns the dependencies between features.

Original: Traditionally, a transformerapplies self-attention to learn context-dependent feature representations.
Revised: Traditionally, a transformer applies self-attention to learn context-dependent feature representations.

Original: Given the success of thisrecipe across various domains, we propose to model gene regulation similarly.
Revised: Given the success of this recipe across various domains, we propose to model gene regulation similarly.

Original: Building on sequence modeling, Exceiver (Expression-Perceiver) is a single-cell gene expressionlanguage model pretrained on an atlas of transcriptomic data.
Revised: Building on sequence modeling, Exceiver (Expression-Perceiver) is a single-cell gene expression language model pretrained on an atlas of transcriptomic data.

Original: We leveraged the Perceiver IOframework to train a long-context sequence model on all protein-coding genes in a self-supervisedmanner [5, 6].
Revised: We leveraged the Perceiver IO framework to train a long-context sequence model on all protein-coding genes in a self-supervised manner [5, 6].

Original: We evaluated latent sample representations with respect to metadata labels includingcell compartment, tissue, and cell ontology.
Revised: We evaluated latent sample representations with respect to metadata labels including cell compartment, tissue, and cell ontology.

Original: We analyzed the similarity of learned gene embeddingsrelative to known molecular interactions.
Revised: We analyzed the similarity of learned gene embeddings relative to known molecular interactions.

Original: Finally, we assessed pretrained Exceiver models on newdatasets and a downstream task.
Revised: Finally, we assessed pretrained Exceiver models on new datasets and a downstream task.

Original: Exceiver provides a framework to learn gene regulatory logic fromunlabeled single-cell transcriptomes and transfer knowledge to new domains.
Revised: Exceiver provides a framework to learn gene regulatory logic from unlabeled single-cell transcriptomes and transfer knowledge to new domains.

Original:  Exceiver accounts for discrete features and technical dropout in scRNA-seqself-supervised pretraining
Revised:  Exceiver accounts for discrete features and technical dropout in scRNA-seq self-supervised pretraining

Original: Exceiver builds on Perceiver IO to encode single-cell transcriptomic profiles.
Revised: Exceiver builds on Perceiver IO to encode single-cell transcriptomic profiles.

Original: Perceiver IO scaleslinearly with the size of inputs and outputs, allowing tractable transformer-based encoding anddecoding of long-context sequences.
Revised: Perceiver IO scales linearly with the size of inputs and outputs, allowing tractable transformer-based encoding and decoding of long-context sequences.

Original: Exceiver retains the core Perceiver IO architectural components:a cross-attention encoder, a self-attention latent process module, and a cross-attention decoder (Figure1a; Methods).
Revised: Exceiver retains the core Perceiver IO architectural components: a cross-attention encoder, a self-attention latent process module, and a cross-attention decoder (Figure 1a; Methods).

Original: Exceiver extends this general architecture to accommodate various biological and experimentalpriors.
Revised: Exceiver extends this general architecture to accommodate various biological and experimental priors.

Original: To account for discrete RNA abundances, Exceiver represents individual genes as learnableembeddings.
Revised: To account for discrete RNA abundances, Exceiver represents individual genes as learnable [Table] embeddings.

Original: Global gene embedding vectors are scaled by expression values to incorporate observed
Revised: Global gene embedding vectors are scaled by expression values to incorporate observed

Original: RNA counts.
Revised: RNA counts.

Original: Exceiver can also augment the learning processes with prior knowledge throughthe integration of experimental metadata.
Revised: Exceiver can also augment the learning processes with prior knowledge through the integration of experimental metadata.

Original: An auxiliary classification option encodes sample classlabels, such as tissue type, as an additional embedding.
Revised: An auxiliary classification option encodes sample class labels, such as tissue type, as an additional embedding.

Original: In addition to biological priors, the Exceiverframework acknowledges that technical dropout influences scRNA-seq experimental measurements.
Revised: In addition to biological priors, the Exceiver framework acknowledges that technical dropout influences scRNA-seq experimental measurements.

Original: Exceiver masks attention computation at these values to mitigate feature sparsity biases in the learningprocess (Figure 1a; Methods).
Revised: Exceiver masks attention computation at these values to mitigate feature sparsity biases in the learning process (Figure 1a; Methods).

Original: To leverage unlabeled single-cell atlases for pretraining, we propose discrete noise masking (DNM),a novel self-supervised task for count-based RNA abundances.
Revised: To leverage unlabeled single-cell atlases for pretraining, we propose discrete noise masking (DNM), a novel self-supervised task for count-based RNA abundances.

Original: DNM randomly chooses a fraction of[Table] genes to mask each time a cell is sampled for training.
Revised: DNM randomly chooses a fraction of genes to mask each time a cell is sampled for training.

Original: A mask embedding replaces sampled geneembeddings and a noised expression value replaces the true expression value.
Revised: A mask embedding replaces sampled gene embeddings and a noised expression value replaces the true expression value.

Original: In our experiments
Revised: In our experiments

Original: DNM simply samples the mean of the feature distribution for expression noising; however, DNMmay extend to other distributions.
Revised: DNM simply samples the mean of the feature distribution for expression noising; however, DNM may extend to other distributions.

Original: (Figure 1a; Methods).
Revised: (Figure 1a; Methods).

Original:  Exceiver learns representations that reflect known biology
Revised:  Exceiver learns representations that reflect known biology

Original: We trained Exceiver on the Tabula Sapiens, a healthy human atlas of approximately 500,000 singlecells from 24 organs of 15 individuals [10].
Revised: We trained Exceiver on the Tabula Sapiens, a healthy human atlas of approximately 500,000 single cells from 24 organs of 15 individuals [10].

Original: We randomly split the data into 70% training and 30%validation sets and pretrained Exceiver with DNM and tissue identity as an auxiliary task.
Revised: We randomly split the data into 70% training and 30% validation sets and pretrained Exceiver with DNM and tissue identity as an auxiliary task.

Original: The modelconverged to an explained variance (EV) of approximately 0.73 and a classification accuracy of 0.61on the validation dataset.
Revised: The model converged to an explained variance (EV) of approximately 0.73 and a classification accuracy of 0. on the validation dataset.

Original: To assess whether Exceiver learned sample representations that reflect biological relationships, weevaluated latent representation similarity relative to that of metadata annotations.
Revised: To assess whether Exceiver learned sample representations that reflect biological relationships, we evaluated latent representation similarity relative to that of metadata annotations.

Original: Qualitatively, boththe original data and Exceiver latent representations reflected tissue of origin and cell compartment(evolutionary lineage) (Figure 1b,c).
Revised: Qualitatively, both the original data and Exceiver latent representations reflected tissue of origin and cell compartment (evolutionary lineage) (Figure 1b,c).

Original: We applied k-means clustering to samples and computed theadjusted mutual information (AMI) between derived cluster labels and true labels.
Revised: We applied k-means clustering to samples and computed the adjusted mutual information (AMI) between derived cluster labels and true labels.

Original: Exceiver latentrepresentations achieved a considerably higher AMI than original samples by tissue, donor, andcompartment labels (Table 1).
Revised: Exceiver latent representations achieved a considerably higher AMI than original samples by tissue, donor, and compartment labels (Table 1).

Original: Given its role as the auxiliary classification task, we expected latentrepresentations to cluster by tissue.
Revised: Given its role as the auxiliary classification task, we expected latent representations to cluster by tissue.

Original: However, structure also increased for donor and compartmentlabels.
Revised: However, structure also increased for donor and compartment labels.

Original: Cell ontology, a fine-grained label, saw decreased clustering relative to original samples.
Revised: Cell ontology, a fine-grained label, saw decreased clustering relative to original samples.

Original: Exceiver’s learned sample representations reflected known relationships and led us to interrogatesimilar structure in learned gene embeddings.
Revised: Exceiver’s learned sample representations reflected known relationships and led us to interrogate similar structure in learned gene embeddings.

Original: We hypothesized that similarity profiles of learned gene embeddings may reflect known gene associations.
Revised: We hypothesized that similarity profiles of learned gene embeddings may reflect known gene associations.

Original: To analyze learned gene relationships, we extracted the vocabulary of global gene embeddingsand applied unsupervised Leiden clustering (Figure 2a).
Revised: To analyze learned gene relationships, we extracted the vocabulary of global gene embeddings and applied unsupervised Leiden clustering (Figure 2a).

Original: Then, we queried the STRING database withthe gene list from each cluster and calculated network enrichment [9].
Revised: Then, we queried the STRING database with the gene list from each cluster and calculated network enrichment [9].

Original: 66% (39/59) of Exceiver geneclusters had more interactions than expected by chance (Figure 2b).
Revised: 66% (39/59) of Exceiver gene clusters had more interactions than expected by chance (Figure 2b).

Original: Additionally, we investigatedfunctional gene associations through Gene Ontology (GO).
Revised: Additionally, we investigated functional gene associations through Gene Ontology (GO).

Original: For example, cluster 17 reflected significant enrichment of GO process terms associated with muscle function (Figure 2c, Figure S1a).
Revised: For example, cluster 17 reflected significant enrichment of GO process terms associated with muscle function (Figure 2c, Figure S1a).

Original: Inanother case, an extremely enriched cluster contained a large portion of ribosomal genes (Figure S1b).
Revised: In another case, an extremely enriched cluster contained a large portion of ribosomal genes (Figure S1b).

Original: Overall, Exceiver gene embeddings reflected network biology associations and captured functionalgene relationships.
Revised: Overall, Exceiver gene embeddings reflected network biology associations and captured functional gene relationships.

Original:  Pretrained Exceiver transfers to new datasets and predicts drug response
Revised:  Pretrained Exceiver transfers to new datasets and predicts drug response

Original: Next, we evaluated the transferability of a pretrained Exceiver model to a new dataset.
Revised: Next we evaluated the transferability of a pretrained Exceiver model to a new dataset.

Original: We expectedthat, compared to an untrained Exceiver model, a pretrained model would more rapidly converge tohigher performance on a new dataset.
Revised: We expected that, compared to an untrained Exceiver model, a pretrained model would more rapidly converge to higher performance on a new dataset.

Original: We tested this on a scRNA-seq dataset generated at a differenttime, in a different lab, of a novel disease physiology.
Revised: We tested this on a scRNA-seq dataset generated at a different time, in a different lab, of a novel disease physiology.

Original: Bi et al.
Revised: Bi et al.

Original: investigated tumor and immunecell reprogramming of patients treated with immune checkpoint blockade for metastatic renal cellcarcinoma [1].
Revised: investigated tumor and immune cell reprogramming of patients treated with immune checkpoint blockade for metastatic renal cell carcinoma [1].

Original: Biopsies from 8 patients were characterized by scRNA-seq for 35,000 cells.
Revised: Biopsies from 8 patients were characterized by scRNAseq for 35,000 cells.

Original: We trained five randomly-initialized Exceiver models on this dataset with DNM and no auxiliary task.
Revised: We trained five randomly-initialized Exceiver models on this dataset with DNM and no auxiliary task.

Original: We likewise fine-tuned five pretrained Exceiver models.
Revised: We likewise fine-tuned five pretrained Exceiver models.

Original: Prior to fine-tuning, the pretrained Exceivermodels predicted masked expression values with an average EV of 0.52.
Revised: Prior to fine-tuning, the pretrained Exceiver models predicted masked expression values with an average EV of 0.52.

Original: The models then convergedto an average EV of 0.94 in under 10 epochs.
Revised: The models then converged to an average EV of 0.94 in under 10 epochs.

Original: This is in contrast to the baseline models, whichconverged to an average EV of 0.73 over a 30x longer training interval ( 350 epochs) (Figure 3).
Revised: This is in contrast to the baseline models, which converged to an average EV of 0. over a 30x longer training interval ( 350 epochs) (Figure 3).

Original: An Exceiver model pretrained on the self-supervised DNM task learns information that is sufficientlygeneral to apply across new datasets and biological contexts.a new dataset.
Revised: An Exceiver model pretrained on the self-supervised DNM task learns information that is sufficiently general to apply across new datasets and biological contexts.

Original: Validation loss and explained variance curves of baseline and pretrained Exceiver models.

Original: Models were trained and evaluated with DNM

Original: Additionally, we hypothesized that a pretrained Exceivermodel would support transfer learning to a new downstream task.
Revised: Additionally, we hypothesized that a pretrained Exceiver model would support transfer learning to a new downstream task.

Original: We turned to a MIX-Seq dataset of pooledcell lines that were transcriptionally characterized afterdrug treatments.
Revised: We turned to a MIX-Seq dataset of pooled cell lines that were transcriptionally characterized after drug treatments.

Original: As in the original study, we matcheddrug response (quantified by area under the dose-responsecurve) from the Genomics of Drug Sensitivity in Cancer Project (GDSC) screen to MIX-Seq transcriptionalprofiles [7, 11].
Revised: As in the original study, we matched drug response (quantified by area under the dose-response curve) from the Genomics of Drug Sensitivity in Cancer Project (GDSC) screen to MIX-Seq transcriptional profiles [7, 11].

Original: We trained five baseline and pretrained Exceiver models on each drug-treated cell line pool (Methods).
Revised: We trained five baseline and pretrained Exceiver models on each drug-treated cell line pool (Methods).

Original: Consistent with the original study, neither approachlearned a relationship between post-perturbation transcriptional profiles and drug responses for navitoclax, taselisib,bortezomib, or gemcitabine, likely due to matching drugresponses from an entirely different dataset.
Revised: Consistent with the original study, neither approach learned a relationship between post-perturbation transcriptional profiles and drug responses for navitoclax, taselisib, bortezomib, or gemcitabine, likely due to matching drug responses from an entirely different dataset.

Original: By contrast,trametinib and dabrafenib treated cell line pools matchedthe most samples and here the pretrained Exceiver succeeded whereas the baseline Exceiver did not ( p < 5e-2,
Revised: By contrast, trametinib and dabrafenib treated cell line pools matched the most samples and here the pretrained Exceiver succeeded whereas the baseline Exceiver did not ( p < 0 .

Original:  Mann-Whitney U; Table 2).
Revised: 05 , Mann-Whitney U; Table 2).

Original: Given the high training variances of the baseline models, further optimization may bewarranted, but it is clear that the pretrained Exceiver modelwas the more robust learner in this challenging scenario. 
Revised: Given the high training variances of the baseline models, further optimization may be warranted, but it is clear that the pretrained Exceiver model was the more robust learner in this challenging scenario. [Table]

Original:  Discussion
Revised:  Discussion

Original: We present Exceiver, a single-cell gene expression language model, whose attention-based transformer backbone encodes long-context transcriptomic profiles.
Revised: We present Exceiver, a single-cell gene expression language model, whose attention-based transformer backbone encodes long-context transcriptomic profiles.

Original: Weintroduce discrete noise masking, a procedure that masksexpression values and enables self-supervised learning onunlabeled, continuously-valued datasets.
Revised: We introduce discrete noise masking, a procedure that masks expression values and enables self-supervised learning on unlabeled, continuously-valued datasets.

Original: We show that an Exceiver model trained on the Tabula Sapiens with a self-supervised task learns low dimensional representations that reflect sample annotations.
Revised: We show that an Exceiver model trained on the Tabula Sapiens with a self-supervised task learns low dimensional representations that reflect sample annotations.

Original: Moreover, learned gene embedding similarity reflects molecular network interactions and functionalassociations.
Revised: Moreover, learned gene embedding similarity reflects molecular network interactions and functional associations.

Original: Finally, we find that a pretrained Exceiver model transfers to new datasets and a drug[Table] response prediction task.
Revised: Finally, we find that a pretrained Exceiver model transfers to new datasets and a drug response prediction task.

Original: Exceiver provides a framework to leverage publicly available scRNA-seqdatasets and learn robust gene regulatory logic across diverse biological contexts.
Revised: Exceiver provides a framework to leverage publicly available scRNA-seq datasets and learn robust gene regulatory logic across diverse biological contexts.

Original: Exceiver mayprovide utility in transferring systems knowledge to downstream tasks, from the interrogation ofmolecular functions to the prediction of comprehensive phenotypes.
Revised: Exceiver may provide utility in transferring systems knowledge to downstream tasks, from the interrogation of molecular functions to the prediction of comprehensive phenotypes.

Original:  Methods 
Revised:  Methods 6.1 Model architecture

Original: 6.1 Model architecture

Original: Perceiver IO is a general purpose model architecture that adapts to any task with structured inputand output.
Revised: Perceiver IO is a general purpose model architecture that adapts to any task with structured input and output.

Original: Since self-attention complexity scales quadratically with input size, it cannot be directlyapplied to high-dimensional data, such as scRNA-seq readouts.
Revised: Since self-attention complexity scales quadratically with input size, it cannot be directly applied to high-dimensional data, such as scRNA-seq readouts.

Original: Perceiver IO addresses this issueby introducing cross-attentional encoder and decoder mechanisms that project to and from a lowerdimensional latent space where full self-attention can be applied.
Revised: Perceiver IO addresses this issue by introducing cross-attentional encoder and decoder mechanisms that project to and from a lowerdimensional latent space where full self-attention can be applied.

Original: Each of the blocks in the Perceiver
Revised: Each of the blocks in the Perceiver

Original: IO architecture are transformer-style modules characterized by query-key-value attention followedby a multilayer perceptron (MLP) and residual connections.
Revised: IO architecture are transformer-style modules characterized by query-key-value attention followed by a multilayer perceptron (MLP) and residual connections.

Original: For more details please reference theoriginal Perceiver IO manuscript [5].
Revised: For more details please reference the original Perceiver IO manuscript [5].

Original: Exceiver implicitly models gene expression as a discrete variable sampled from a count-baseddistribution.
Revised: Exceiver implicitly models gene expression as a discrete variable sampled from a count-based distribution.

Original: Global embeddings are used to represent gene identities, which are scaled by expressionvalues upon model input.
Revised: Global embeddings are used to represent gene identities, which are scaled by expression values upon model input.

Original: Exceiver also accounts for dropout, a sequencing bias characterized byexperimental failure to capture the comprehensive set of genes expressed in an individual cell.
Revised: Exceiver also accounts for dropout, a sequencing bias characterized by experimental failure to capture the comprehensive set of genes expressed in an individual cell.

Original: Thislimitation implies that unobserved genes may be either truly unexpressed or merely unmeasured.
Revised: This limitation implies that unobserved genes may be either truly unexpressed or merely unmeasured.

Original: Exceiver masks the attention computation at zero-valued expression positions to prevent learningbased on this artifact.
Revised: Exceiver masks the attention computation at zero-valued expression positions to prevent learning based on this artifact.

Original: Upon decoding, Exceiver reuses unscaled gene embeddings in the outputquery matrix as a means to residually connect discrete gene identities within the learning process.
Revised: Upon decoding, Exceiver reuses unscaled gene embeddings in the output query matrix as a means to residually connect discrete gene identities within the learning process.

Original: Exceiver also provides an option to bias learning with an auxiliary classification task.
Revised: Exceiver also provides an option to bias learning with an auxiliary classification task.

Original: An additionalclassification embedding token is concatenated to each input sequence and used to query the decoder.
Revised: An additional classification embedding token is concatenated to each input sequence and used to query the decoder.

Original: An auxiliary MLP classifies the output of the decoder from the classification token, and an additionalcross-entropy loss is added to the mean squared error from the DNM task.
Revised: An auxiliary MLP classifies the output of the decoder from the classification token, and an additional cross-entropy loss is added to the mean squared error from the DNM task.

Original: Code is available athttps://github.com/keiserlab/exceiver .
Revised: Code is available at https://github.com/keiserlab/exceiver .

Original: 6.2 Pretraining tasks
Revised: 6.2 Pretraining tasks

Original: Exceiver employs a self-supervised pretraining task as well as an optional auxiliary supervisedclassification task.
Revised: Exceiver employs a self-supervised pretraining task as well as an optional auxiliary supervised classification task.

Original: The self-supervised task draws inspiration from masking tasks applied for NLPpretraining.
Revised: The self-supervised task draws inspiration from masking tasks applied for NLP pretraining.

Original: However, there are several crucial differences between gene expression profiles andnatural language.
Revised: However, there are several crucial differences between gene expression profiles and natural language.

Original: First, scRNA-seq features are a set rather than a sequence.
Revised: First, scRNA-seq features are a set rather than a sequence.

Original: Exceiver does notpositionally encode gene embeddings, though features that describe spatial gene dependencies (suchas the linear position of a gene or its relative position in the three-dimensional chromosome) mayprove valuable in future work.
Revised: Exceiver does not positionally encode gene embeddings, though features that describe spatial gene dependencies (such as the linear position of a gene or its relative position in the three-dimensional chromosome) may prove valuable in future work.

Original: Second, as previously referenced, expression features are counts ofdiscrete features (genes), rather than identities of discrete features (words).
Revised: Second, as previously referenced, expression features are counts of discrete features (genes), rather than identities of discrete features (words).

Original: As such, pretraining isa regression task rather than a classification task.
Revised: As such, pretraining is a regression task rather than a classification task.

Original: Exceiver employs a new self-supervised task toaccount for the discrete distributions of features in the gene set.
Revised: Exceiver employs a new self-supervised task to account for the discrete distributions of features in the gene set.

Original: The primary pretraining task we apply is discrete noise masking (DNM).
Revised: The primary pretraining task we apply is discrete noise masking (DNM).

Original: First, we heuristicallydetermine the number of genes to mask for each sample.
Revised: First, we heuristically determine the number of genes to mask for each sample.

Original: We mask 15% of the median number ofrecorded genes across the training dataset.
Revised: We mask 15% of the median number of recorded genes across the training dataset.

Original: We avoid masking a majority fraction of genes capturedin poorly sampled cells by removing cells below a minimum number of recorded genes.
Revised: We avoid masking a majority fraction of genes captured in poorly sampled cells by removing cells below a minimum number of recorded genes.

Original: In ourexperiments, we set this threshold at twice the number of masked genes.
Revised: In our experiments, we set this threshold at twice the number of masked genes.

Original: The training data thereforereflects a distribution of masked genes not exceeding 50% of the observed genes in a given cell (Figure
Revised: The training data therefore reflects a distribution of masked genes not exceeding 50% of the observed genes in a given cell (Figure

Original: S2).
Revised: S2).

Original: A “mask” embedding replaces the randomly selected gene embeddings, and correspondingexpression values are noised.
Revised: A “mask” embedding replaces the randomly selected gene embeddings, and corresponding expression values are noised.

Original: Masked gene embeddings scaled by noised expression values passthrough encoder and process modules.
Revised: Masked gene embeddings scaled by noised expression values pass through encoder and process modules.

Original: Finally, the embeddings of masked genes query the decoder.
Revised: Finally, the embeddings of masked genes query the decoder.

Original: The resulting “contextual gene embeddings” pass through the final MLP, outputting expressionpredictions for masked genes (Figure 1a).
Revised: The resulting “contextual gene embeddings” pass through the final MLP, outputting expression predictions for masked genes (Figure 1a).

Original: Additionally, we propose an auxiliary classification task based on metadata labels.
Revised: Additionally, we propose an auxiliary classification task based on metadata labels.

Original: The Perceiver IOarchitecture is flexible enough to accommodate multimodal tasks.
Revised: The Perceiver IO architecture is flexible enough to accommodate multimodal tasks.

Original: Exceiver implements an auxiliaryclassification task as described in the original paper.
Revised: Exceiver implements an auxiliary classification task as described in the original paper.

Original: A classification token is initialized as a globalembedding and passed through Exceiver with each gene set.
Revised: A classification token is initialized as a global embedding and passed through Exceiver with each gene set.

Original: This procedure allows for attentioncomputation between all genes and the classification token.
Revised: This procedure allows for attention computation between all genes and the classification token.

Original: The original classification embedding queries the decoder along with masked genes, and the resulting “contextual classification embedding”is passed through an auxiliary MLP head, outputting a vector of class logits.
Revised: The original classification embedding queries the decoder along with masked genes, and the resulting “contextual classification embedding” is passed through an auxiliary MLP head, outputting a vector of class logits.

Original: 6.3 Data and processing
Revised: 6.3 Data and processing

Original: The Tabula Sapiens (TS) data was retrieved from figshare ( https://figshare.com/ndownloader/files/34702114 ).
Revised: The Tabula Sapiens (TS) data was retrieved from figshare ( https://figshare.com/ ndownloader/files/34702114 ).

Original: Python packages anndata and scanpy were used for data storageand statistics calculations.
Revised: Python packages anndata and scanpy were used for data storage and statistics calculations.

Original: Features were subsetted to all protein-coding genes as defined in the
Revised: Features were subsetted to all protein-coding genes as defined in the

Original: Cancer Cell Line Encyclopedia (CCLE) ( n = 19067 ).
Revised: Cancer Cell Line Encyclopedia (CCLE) ( n = 19067 ).

Original: DecontX-corrected count matrices wereused for training to account for ambient RNA.
Revised: DecontX-corrected count matrices were used for training to account for ambient RNA.

Original: TS was shuffled and split into 70% training and 30%validation datasets.
Revised: TS was shuffled and split into 70% training and 30% validation datasets.

Original: Counts were normalized to a maximum of 1 e 4 per cell.
Revised: Counts were normalized to a maximum of 1 e 4 per cell.

Original: Normalized countswere log-transformed and scikit-learn was used to apply a Z-score transformation to each datasetindividually.
Revised: Normalized counts were log-transformed and scikit-learn was used to apply a Z-score transformation to each dataset individually.

Original: Expression values were shifted to a mean-center of 1, implying the learned identity ofglobal gene embeddings corresponds to average expression.
Revised: Expression values were shifted to a mean-center of 1, implying the learned identity of global gene embeddings corresponds to average expression.

Original: The Bi et al.
Revised: The Bi et al.

Original: data was retrieved from the Human Cell Atlas collection ( https://singlecell.broadinstitute.org/single_cell/study/SCP1288 ).
Revised: data was retrieved from the Human Cell Atlas collection ( https://singlecell. broadinstitute.org/single_cell/study/SCP1288 ).

Original: Raw count matrices were split into 70%training and 30% validation, and the same normalization and transformation procedure was applied.
Revised: Raw count matrices were split into 70% training and 30% validation, and the same normalization and transformation procedure was applied.

Original: Missing genes were imputed with a value of zero.
Revised: Missing genes were imputed with a value of zero.

Original: The MIX-Seq data from McFarland et al.
Revised: The MIX-Seq data from McFarland et al.

Original: was retrieved from figshare ( https://figshare.com/s/139f64b495dea9d88c70 ).
Revised: was retrieved from figshare ( https://figshare.com/s/ 139f64b495dea9d88c70 ).

Original: Sanger GDSC2 AUC data was downloaded from the Dependency Map( https://depmap.org/portal/download/custom/ ).
Revised: Sanger GDSC2 AUC data was downloaded from the Dependency Map ( https://depmap.org/portal/download/custom/ ).

Original: Drugs from 24-hr treatment experimentswith sufficient matching GDSC2 experiments were selected for analysis.
Revised: Drugs from 24-hr treatment experiments with sufficient matching GDSC2 experiments were selected for analysis.

Original: Post-perturbation transcriptome datasets were preprocessed individually in the manner as described previously with zero-valuemissing geneimputation. Cell line-drug transcriptome pairs were matched to dose-response curve AUC values of [Table]
Revised: Post-perturbation transcriptome datasets were preprocessed individually in the manner as described previously with zero-value missing gene

Original: [Figure]

Original: Figures