Original: PMI- GUIDED M ASKING S TRATEGY TO E NABLE F EWSHOT L EARNING FOR G ENOMIC A PPLICATIONS
Revised: PMI- GUIDED M ASKING S TRATEGY TO E NABLE F EWSHOT L EARNING FOR G ENOMIC A PPLICATIONS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Learning effective gene representations is of great research interest.
Revised: Learning effective gene representations is of great research interest.

Original: Lately, largescale language models based on the transformer architecture, such as DNABert and LOGO, have been proposed to learn gene representations from the Human
Revised: Lately, largescale language models based on the transformer architecture, such as DNABert and LOGO, have been proposed to learn gene representations from the Human

Original: Reference Genome .
Revised: Reference Genome .

Original: Although these large language models outperform previous approaches, currently, no study empirically determined the best strategy for representing gene sequences as tokens.
Revised: Although these large language models outperform previous approaches, currently, no study empirically determined the best strategy for representing gene sequences as tokens.

Original: The uniform random masking strategy, which is the default during the pretraining of such masked language models, might lead to pretraining inefﬁciency, resulting in suboptimal downstream task performance in the few-shot setting.
Revised: The uniform random masking strategy, which is the default during the pretraining of such masked language models, might lead to pretraining inefficiency, resulting in suboptimal downstream task performance in the few-shot setting.

Original: However, good few-shot performance is critical, with dataset sizes in (personalized) medicine often not exceeding a couple of hundred data points.
Revised: However, good few-shot performance is critical, with dataset sizes in (personalized) medicine often not exceeding a couple of hundred data points.

Original: In this paper, we develop a novel strategy to adapt Pointwise Mutual Information (PMI) masking used previously in the NLP setting to the domain of gene sequence modeling.
Revised: In this paper, we develop a novel strategy to adapt Pointwise Mutual Information (PMI) masking used previously in the NLP setting to the domain of gene sequence modeling.

Original: PMI-masking masks spans of tokens that are more likely to co-occur, forming a statistically relevant span.
Revised: PMI-masking masks spans of tokens that are more likely to co-occur, forming a statistically relevant span.

Original: First, we learn a vocabulary of tokens with a high PMI score from our pretraining corpus (the Human Reference
Revised: First, we learn a vocabulary of tokens with a high PMI score from our pretraining corpus (the Human Reference

Original: Genome ).
Revised: Genome ).

Original: Next, we utilize this side information (pre)train our model by masking tokens based on PMI scores.
Revised: Next, we utilize this side information (pre)train our model by masking tokens based on PMI scores.

Original: In extensive experiments, we evaluate the effectiveness of the PMI-masking strategy on two baseline models of DNABert and LOGO, over three benchmark datasets (two on promoters and one on enhancer), and on a variety of few-shot settings.
Revised: In extensive experiments, we evaluate the effectiveness of the PMI-masking strategy on two baseline models of DNABert and LOGO, over three benchmark datasets (two on promoters and one on enhancers), and on a variety of few-shot settings.

Original: We observe that our PMI-masking-guided baseline models substantially outperform the SOTA models.
Revised: We observe that our PMI-masking-guided baseline models substantially outperform the SOTA models.

Original: We further observe that almost all the top-ranked DNA tokens in terms of PMI score are closely associated with existing conserved DNA sequence motifs .
Revised: We further observe that almost all the top-ranked DNA tokens in terms of PMI score are closely associated with existing conserved DNA sequence motifs .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Computational analysis of genomics has revolutionized the ﬁeld of medical science (McGuire et al., 2020), particularly with the advent of the Human Reference Genome (Schneider et al., 2016).
Revised: Computational analysis of genomics has revolutionized the field of medical science (McGuire et al., 2020), particularly with the advent of the Human Reference Genome (Schneider et al., 2016).

Original: As seen in (Yue & Wang, 2018a), deep learning has been applied to various applications, such as protein structure analysis, gene expression data, and transcriptome analysis.
Revised: As seen in (Yue & Wang, 2018a), deep learning has been applied to various applications, such as protein structure analysis, gene expression data, and transcriptome analysis.

Original: Given the sequential nature of gene sequences, several deep learning models found to be effective in the Natural Language Processing (NLP) domain have been adopted for genomic applications (Yue & Wang, 2018b).
Revised: Given the sequential nature of gene sequences, several deep learning models found to be effective in the Natural Language Processing (NLP) domain have been adopted for genomic applications (Yue & Wang, 2018b; Avsec et al., 2021).

Original: The input data for these tasks is often presented as a sequence of nucleotides.
Revised: The input data for these tasks is often presented as a sequence of nucleotides.

Original: Each side of the double-helix DNA strand comprises the bases adenine (A), cytosine (C), guanine (G), and thymine (T)).
Revised: Each side of the double-helix DNA strand comprises the bases adenine (A), cytosine (C), guanine (G), and thymine (T)).

Original: However, unlike words or sentences in languages, there are no clear sematically demarcated tokens present within the gene sequence.
Revised: However, unlike words or sentences in languages, there are no clear semantically demarcated tokens present within the gene sequence.

Original: Therefore, to come to a workable solution, researchers (Mo et al., 2021; Ji et al., 2021) use k -mer representation, which is a sliding window of k -length over the entire sequence.
Revised: Therefore, to come to a workable solution, researchers (Mo et al., 2021; Ji et al., 2021) use k -mer representation, which is a sliding window of k -length over the entire sequence.

Original: For example, if the gene sequence is ATTCGATGC , a 6-mer representation will be ATTCGA , TTCGAT , TCGATG and CGATGC .
Revised: For example, if the gene sequence is ATTCGATGC , a 6-mer representation will be ATTCGA , TTCGAT , TCGATG and CGATGC .

Original: Similar to the NLP domain, the standard approach in gene sequence modeling is to pretrain the transformer models by randomly masking (and predicting) tokens, the so-called masked language modeling objective (MLM).
Revised: Similar to the NLP domain, the standard approach in gene sequence modeling is to pretrain the transformer models by randomly masking (and predicting) tokens, the so-called masked language modeling objective (MLM).

Original: Given the lack of rigorous evaluations of whether the methods from the NLP domain transfer to modeling gene sequences, we question whether token selection, masking, or other components of the transformer architecture are restrictive and not grounded in biomedical domain knowledge.
Revised: Given the lack of rigorous evaluations of whether the methods from the NLP domain transfer to modeling gene sequences, we question whether token selection, masking, or other components of the transformer architecture are restrictive and not grounded in biomedical domain knowledge.

Original: We develop a principled statistical approach based on Pointwise Mutual Information (PMI) to automatically identify meaningful spans (tokens) from a DNA sequence.
Revised: We develop a principled statistical approach based on Pointwise Mutual Information (PMI) to automatically identify meaningful spans (tokens) from a DNA sequence.

Original: PMI involves taking both the frequencies of a token and its overall uniqueness in the dataset.
Revised: PMI involves taking both the frequencies of a token and its overall uniqueness in the dataset.

Original: However, the standard PMI formulation (Levine et al., 2020) sometimes favors tokens with  low frequency of occurrence.
Revised: However, the standard PMI formulation (Levine et al., 2021) sometimes favors tokens with a low frequency of occurrence.

Original: To adapt to the genomic setting, we modify the PMI metric to mitigate the above-mentioned issue.
Revised: To adapt to the genomic setting, we modify the PMI metric to mitigate the above-mentioned issue.

Original: The state-of-the-art models (DNABert (Ji et al., 2021) and LOGO (Yang et al., 2021)) in gene regulatory sequence classiﬁcation tasks are widely used in the literature (Yang et al., 2021; Mo et al., 2021;
Revised: The state-of-the-art models (DNABert (Ji et al., 2021) and LOGO (Yang et al., 2021)) in gene regulatory sequence classification tasks are widely used in the literature (Yang et al., 2021; Mo et al., 2021;

Original: Badirli et al., 2021).
Revised: Badirli et al., 2021).

Original: Instead of randomly selecting tokens to mask, as done in DNABert (or LOGO), we use the PMI score to prioritize the relevant tokens to mask within a given gene sequence.
Revised: Instead of randomly selecting tokens to mask, as done in DNABert (or LOGO), we use the PMI score to prioritize the relevant tokens to mask within a given gene sequence.

Original: This strategy is inspired by the work of Levine et al.
Revised: This strategy is inspired by the work of Levine et al.

Original: (2020), which shows that PMI-masking improves over random span masking (Joshi et al., 2020) in NLP tasks.
Revised: (2021), which shows that PMI-masking improves over random span masking (Joshi et al., 2020) in NLP tasks.

Original: However, the main difference from our work is that they use it to develop a large masking vocabulary in the NLP domain, while we use the
Revised: However, the main difference from our work is that they use it to develop a large masking vocabulary in the NLP domain, while we use the

Original: PMI score as a ranking function to choose the span of tokens to mask.
Revised: PMI score as a ranking function to choose the span of tokens to mask.

Revised: Instead of using PMI score to formulate an absolute importance notion and consequently create a masking vocabulary, we use the PMI score to measure relative importance among DNA sequence k-mer tokens during MLM.

Original: As current medical datasets often face data scarcity issues, the move towards personalized medicine require models that perform well with limited training data at hand (Shaikhina & Khovanova, 2017;
Revised: As current medical datasets often face data scarcity issues, the move towards personalized medicine requires models that perform well with limited training data at hand (Shaikhina & Khovanova, 2017;

Original: Hekler et al., 2019).
Revised: Hekler et al., 2019).

Original: We therefore evaluate our proposed PMI-masking strategy in the low-resource (few-shot) setting.
Revised: We, therefore, evaluate our proposed PMI-masking strategy in the low-resource (few-shot) setting.

Original: Introducing biomedical domain knowledge in the form of side information is a promising research direction, speciﬁcally to deal with limited data (Kyono et al., 2019; Oei et al., 2021; Roy et al., 2021).
Revised: Introducing biomedical domain knowledge in the form of side information is a promising research direction, specifically to deal with limited data (Kyono et al., 2019; Oei et al., 2021; Roy et al., 2021).

Original: Our PMI-guided approach builds on that idea and  includes side-information derived from the Human Reference Genomedirectly into the training procedure.
Revised: Our PMI-guided approach builds on that idea and directly includes side information derived from the Human Reference Genome into the training procedure.

Original: Our extensive experimentation shows that PMI-masking-guided DNABert and LOGO improve over the standard random masking-guided DNABert and LOGO, respectively, in few-shot settings ( 10 , 50 , 100 , 500 and 1000 training data points per class) over three benchmark datasets of gene sequence classiﬁcation (two on promoters and one on enhancer).
Revised: Our extensive experimentation shows that PMI-masking-guided DNABert and LOGO improve over the standard random masking-guided DNABert and LOGO (pretrained on the same number of steps) respectively, in few-shot settings ( 10 , 50 , 100 , 500 and 1000 training data points per class) over three benchmark datasets of gene sequence classification (two on promoters such as Prom-core, Promand one on enhancer such as Cohn-enh ). We posit that PMI-masking helps incorporate non-trivial genetic knowledge because we observe that PMI-masking-guided DNABert pretrained for 10K steps even outperforms original DNABert pretrained for 120K steps for all few-shot settings in case of Prom-300 and Cohn-enh dataset.

Original: In addition, we perform motif 1 analysis and ﬁne-tuning impact analysis to understand the domain knowledge learned by PMI-guided DNABert .
Revised: In addition, we perform motif 1 analysis and finetuning impact analysis to understand the domain knowledge learned by PMI-guided DNABert.

Original: We observe that PMI-based rankings capture DNA sequence motifs present in downstream (task) datasets.
Revised: To alleviate the issue of tremendous engineering effort needed to develop the experimental setup of gene sequence modeling, we plan to make the pretraining and finetuning datasets, model weights and checkpoints, and associated codebase publicly available, as the final contribution.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Learning deep representations in the context of gene sequence modeling.
Revised: Learning deep representations in the context of gene sequence modeling.

Original: Nguyen et al.
Revised: Nguyen et al.

Original: (2016) encode base pair triples as one-hot vectors to feed into convolutional neural networks for DNA sequence classiﬁcation tasks, whereas Badirli et al.
Revised: (2016) encode base pair triples as one-hot vectors to feed into convolutional neural networks for DNA sequence classification tasks, whereas Badirli et al.

Original: (2021) convert the DNA barcodes represented by nucleotide sequences into a vector embedding useful for the task of ﬁne-grained species classiﬁcation.
Revised: (2021) convert the DNA barcodes represented by nucleotide sequences into a vector embedding useful for the task of fine-grained species classification.

Original: Ng (2017) utilize k-mers to represent gene sequences in their approach using a shallow neural network.
Revised: Ng (2017) utilize k-mers to represent gene sequences in their approach using a shallow neural network.

Original: Ji et al.
Revised: Ji et al.

Original: ; Yang et al.
Revised: ; Yang et al.

Original: (2021) also represent gene sequences as k-mers, by using a learned dense representation from an adapted BERT model.
Revised: (2021) also represent gene sequences as k-mers, by using a learned dense representation from an adapted BERT model.

Original: Mock et al.
Revised: Mock et al.

Original: (2021) largely adapt the
Revised: (2021) largely adapt the

Original: DNABert architecture for the task of taxonomy classiﬁcation and model gene sequences as 3-mers, but also includes next sentence prediction along with MLM training loss.
Revised: DNABert architecture for the task of taxonomy classification and model gene sequences as 3-mers, but also includes next sentence prediction along with MLM training loss.

Original: Zaheer et al.

Original: (2020) propose the BigBird language model and evaluate it on different long-sequence tasks, one of which is gene sequence modeling.

Original: Instead of using k-mer representations, Zaheer et al.
Revised: Instead of using k-mer representations, Zaheer et al.

Original: (2020) trained a SentencePiece tokenizer on the reference genome, resulting in 32k tokens with an average length of 8.78 basepairs/token.
Revised: (2020) trained a SentencePiece tokenizer on the Human Reference

Original: Their BigBird model was trained on the Human Reference Genome and applied to the tasks of chromatine-proﬁle prediciton and promoter region prediction.
Revised: Genome and applied to the tasks of chromatin-profile prediction and promoter region prediction.

Original: Mo et al.
Revised: Mo et al.

Original: (2021) infuse domain knowledge into the model by proposing a multimodal pretraining setup. In addition to the gene sequences, the authors also add information about transcription factors and regions to the input.
Revised: (2021) infuse domain knowledge into the model by proposing a multimodal pretraining setup comprising gene sequences and information on transcription factors and regions.

Original: Random and PMI-masking for MLM training in NLP.
Revised: Random and PMI-masking for MLM training in NLP.

Original: The initial work applied random token masking , as performed by BERT (Devlin et al., 2018), where 15% of the input tokens are chosen to be masked uniformly.
Revised: The initial work applied random token masking , as performed by BERT (Devlin et al., 2018), where 15% of the input tokens are chosen to be masked uniformly.

Original: Previous work has also investigated jointly masking whole words ( whole word masking (Sennrich et al., 2016;
Revised: Previous work has also investigated jointly masking whole words ( whole word masking (Sennrich et al., 2016;

Original: Devlin et al., 2019)) or entities ( entity masking (Sun et al., 2019)) which both have been shown to be beneﬁcial over masking tokens.
Revised: Devlin et al., 2019)) or entities ( entity masking (Sun et al., 2019)) which both have been shown to be beneficial over masking tokens.

Original: Joshi et al.
Revised: Joshi et al.

Original: (2020) propose random span masking where random spans with lengths chosen from a geometric distribution are masked at random positions.
Revised: (2020) propose random span masking where random spans with lengths chosen from a geometric distribution are masked at random positions.

Original: This simple method has been shown to outperform the more in- volved entity masking approach.
Revised: This simple method has been shown to outperform the more involved entity masking approach.

Original: When applied to gene sequences, especially entity and whole word masking have the problem that there is no well-deﬁned concept of entites and words in gene sequences.
Revised: When applied to gene sequences, especially entity and whole word masking have the problem that there is no well-defined concept of entities and words in gene sequences.

Original: The PMI-masking approach (Levine et al., 2020) builds on the ideas of entity and span masking, but instead of masking random spans, their approach involves measuring the relevance of colocated n-grams (based on their constructed masking vocabulary of 800K tokens) with pointwise mutual information (PMI) and masking these spans together.
Revised: The PMI-masking approach (Levine et al., 2021) builds on the ideas of entity and span masking, but instead of masking random spans, their approach involves measuring the relevance of colocated n-grams (based on their constructed masking vocabulary of 800K tokens) with pointwise mutual information (PMI) and masking these spans together.

Original: The authors show that masking PMI tokens (i) accelerates training while matching end-of-pretraining performance in roughly half the training steps and (ii) improves upon previous masking approaches at the end of pretraining.
Revised: The authors show that masking PMI tokens (i) accelerates training while matching end-of-pretraining performance in roughly half the training steps and (ii) improves upon previous masking approaches at the end of pretraining.

Original:  B UILDING BLOCKS OF SOTA MODELS
Revised:  B UILDING BLOCKS OF SOTA MODELS

Original: We focus on the two recent transformer-based pretrained models of DNABert Ji et al.
Revised: We focus on the two recent transformer-based pretrained models of DNABert Ji et al.

Original: (2021) and
Revised: (2021) and

Original: LOGO Yang et al.
Revised: LOGO Yang et al.

Original: (2021) that are adapted to the gene sequence modeling domain.
Revised: (2021) that are adapted to the gene sequence modeling domain.

Original: Through MLM pre-training, these SOTA models learn powerful contextual representations for DNA fragments utilizing abundant unlabeled data from the Human Reference Genome , which contains around 3.
Revised: Through MLM pre-training, these SOTA models learn powerful contextual representations for DNA fragments utilizing abundant unlabeled data from the Human Reference Genome , which contains around 3.

Original: billion base pairs in total over 24 chromosomes.
Revised: billion base pairs in total over 24 chromosomes.

Original: Implementation details of SOTA models and associated research challenges.
Revised: Implementation details of SOTA models and associated research challenges.

Original: The tokenization of gene sequences and MLM training is performed based on the author’s codebase (DNA, 2021).
Revised: The tokenization of gene sequences and MLM training is performed based on the author’s codebase (DNA, 2021).

Original: However, they do not provide the dataset for pretraining or ﬁnetuning (downstream) tasks.
Revised: However, they do not provide the dataset for pretraining or finetuning (downstream) tasks.

Original: Therefore, we follow the author’s description to construct the corresponding datasets, which is nontrivial.
Revised: Therefore, we follow the author’s description to construct the corresponding datasets, which is nontrivial.

Original: We explain the pretraining data creation process in this section and later describe the ﬁnetuning dataset creation process in Section 5.1.
Revised: We explain the pretraining data creation process in this section and later describe the finetuning dataset creation process in Section 5.1.

Original: Tokenization of gene sequences.
Revised: Tokenization of gene sequences.

Original: The gene sequence is ﬁrst converted into a k-mer representation, which is commonly used in the literature (Ng, 2017; Ji et al., 2021).
Revised: The gene sequence is first converted into a k-mer representation, which is commonly used in the literature (Ng, 2017; Ji et al., 2021).

Original: The k -mer representation is a sliding window of length k .
Revised: The k -mer representation is a sliding window of length k .

Original: For example, AGCACGCAG in 6-mer representation leads to 3 tokens - AGCACG , GCACGA ,and CACGAG .
Revised: For example, AGCACGCAG in 6-mer representation leads to 3 tokens - AGCACG , GCACGA , CACGAG .

Original: Thus, the vocabulary comprises all combinations ( 4 k length) and ﬁve special tokens - CLS, PAD, UNK, SEP, MASK .
Revised: Thus, the vocabulary comprises all combinations ( 4 k length) and five special tokens - CLS, PAD, UNK, SEP, MASK .

Original: According to the set-up chosen by SOTA models, we consider k = 6 for all the experiments Yang et al.
Revised: According to the set-up chosen by SOTA models, we consider k = 6 for all the experiments. Yang et al.

Original: (2021) note that 6-mers incorporate richer contextual information while keeping the  computational complexityand memory requirement manageable.
Revised: (2021) note that 6-mers incorporate richer contextual information while keeping the memory and computational complexity manageable.

Original: Pretraining data preparation.
Revised: Pretraining data preparation.

Original: Since the pretraining data is not provided with the author’s codebase, we follow the implementation details mentioned in the paper to construct the pretraining dataset, which is a non-trivial task.
Revised: Since the pretraining data is not provided with the author’s codebase, we follow the implementation details mentioned in the paper to construct the pretraining dataset, which is a non-trivial task.

Original: We obtain the Human Reference Genome from the Genome
Revised: We obtain the Human Reference Genome from the Genome

Original: Reference Consortium Human Build 38 patch release 13 (GRCh38.p13) FASTA ﬁle Consortium (2019) from the NCBI website.
Revised: Reference Consortium Human Build 38 patch release 13 (GRCh38.p13) FASTA file Consortium (2019) from the NCBI website.

Original: It serves as a large-scale corpus of unlabeled gene sequence data, which we use for MLM training to obtain a contextual representation of the 6-mer tokens.
Revised: It serves as a large-scale corpus of unlabeled gene sequence data, which we use for MLM training to obtain a contextual representation of the 6-mer tokens.

Original: We perform the following steps to convert the Human Reference Genome to a form that DNABert (or
Revised: We perform the following steps to convert the Human Reference Genome to a form that DNABert (or

Original: LOGO) can use to train with the MLM objective:
Revised: LOGO) can use to train with the MLM objective:

Original: Step 1: For each chromosome c in the Human Reference Genome, we randomly choose the starting index between 1 and 1000 Zaheer et al.
Revised: Step 1: For each chromosome c in the Human Reference Genome, we randomly choose the starting index between 1 and 1000 Zaheer et al.

Original: Step 2:
Revised: Step 2:

Original: Given the chromosome number and its starting index ( ST ), we next determine the length of the DNA segment L as BERT has the limitation of accommodating a maximum of 512 tokens.
Revised: Given the chromosome number and its starting index ( ST ), we next determine the length of the DNA segment L as BERT has the limitation of accommodating a maximum of 512 tokens.

Original: We select L as 510 for 50% of the cases and a randomly selected length between 5 and 510 for the remaining 50% of cases Ji et al.
Revised: We select L as 510 for 50% of the cases and a randomly selected length between 5 and 510 for the remaining 50% of cases Ji et al.

Original: Step 3: We thus create the DNA segment comprising the base pairs between ST and ST + L of chromosome c , corresponding to data point in the pretraining dataset .
Revised: Step 3: We thus create the DNA segment comprising the base pairs between ST and ST + L of chromosome c , corresponding to data point in the pretraining dataset .

Original: We ﬁlter out DNA segments that contain bases other than A, T, C, or G.
Revised: We filter out DNA segments that contain bases other than A, T, C, or G.

Original: MLM training.
Revised: MLM training.

Original: The SOTA models are trained with masked language modeling loss similar to
Revised: The SOTA models are trained with masked language modeling loss similar to

Original: BERT Devlin et al.
Revised: BERT Devlin et al.

Original: However, to mask a nucleotide, a contiguous sequence of tokens is masked to prevent information leakage, as each nucleotide is part of k consecutive k-mers.
Revised: However, to mask a nucleotide, a contiguous sequence of tokens is masked to prevent information leakage, as each nucleotide is part of k consecutive k-mers.

Original: More formally, say a neucleotide is represented a DNA[ i ], while a 6-mer token is represented as T 

Revised: More formally, say a neucleotide is represented a DNA[ i ], while a 6-mer token is represented as T

Original: [ i ] =
Revised: [ i ] =

Original: { DNA[ i -2], DNA[ i - 1] · · · DNA[ i + 3] } , then the tokens T[ j ], ∀ ( j ) i
Revised: { DNA[ i -2], DNA[ i - 1] · · · DNA[ i + 3] } , then the tokens T[ j ], ∀ ( j ) i

Original: +

Revised: +

Original: j = i − 3 is masked.
Revised: j = i − 3 is masked.

Original: Given that k = 6 and 15% of tokens need to be masked (Devlin et al., 2018), the MLM probability is set at: 15% / 6 ; that is, 2 . 5% of the nucleotides are chosen for masking.
Revised: Given that k = 6 and 15% of tokens need to be masked (Devlin et al., 2018), the MLM probability is set at: 15% / 6 ; that is, 2 . 5% of the nucleotides are chosen for masking.

Original:  P ROPOSED A DAPTATION OF PMI- MASKING S TRATEGY TO G ENOMIC A PPLICATIONS
Revised:  P ROPOSED A DAPTATION OF PMI-M ASKING S TRATEGY TO G ENOMIC A PPLICATIONS

Original: In this paper, we use a PMI-based approach to identify spans of k-mers that co-occur much more than expected compared with their components (i.e., k-mers of shorter length such as 4-mers or 5-mers).
Revised: In this paper, we use a PMI-based approach to identify spans of k-mers that co-occur much more than expected compared to their components (i.e., k-mers of shorter length, such as 4-mers or 5-mers).

Original: These spans then replace the (uniform) random masking strategy used by SOTA models such as
Revised: These spans then replace the (uniform) random masking strategy used by SOTA models such as

Original: DNABert and LOGO.
Revised: DNABert and LOGO.

Original: This is to make the masked token prediction task more difﬁcult by removing highly correlated local contexts, which subsequently may improve the pretraining efﬁciency (as shown by Levine et al.
Revised: This is to make the masked token prediction task more difficult by removing highly correlated local contexts, which subsequently may improve the pretraining efficiency (as shown by Levine et al.

Original: In this section, we discuss the novel strategy to ﬁrst adapt the PMI metric from NLP to the genomic setting (Section 4.1), and then explain how to use PMI-masking for improving the efﬁciency of MLM training (Section 4.2).
Revised: In this section, we discuss the novel strategy to first adapt the PMI metric from NLP to the genomic setting (Section 4.1) and then explain how to use PMI-masking for improving the efficiency of MLM training (Section 4.2).

Original: 4. PMI SCORING FOR G ENE S EQUENCES
Revised: 4. PMI S CORING FOR G ENE S EQUENCES

Original: Although NLP models are applied to the biomedical domain, no study empirically determined the best strategy for representing gene sequences.
Revised: Although NLP models are applied to the biomedical domain, no study empirically determined the best strategy for representing gene sequences.

Revised: In this work, we consider a single nucleotide equivalent to a single token in NLP, and thus an n-gram from the NLP domain is equivalent to a k-mer from the gene sequence modeling literature.

Original: We propose that PMI-based scoringcan be adapted to our genomic setting and can help us identify high PMI tokens to mask.
Revised: We propose a novel strategy to adapt PMI-based scoring to our genomic setting and can help us identify high PMI tokens to mask.

Original: Pointwise Mutual Information (PMI) quantiﬁes how often two tokens occur compared to what is expected if they are independent.
Revised: Pointwise Mutual Information (PMI) quantifies how often two tokens occur compared to what is expected if they are independent.

Original: The PMI formula (proposed by Levine et al.
Revised: The PMI formula (proposed by Levine et al.

Original: (2020)) when extended to n-grams (where n > 2) is:
Revised: (2021)) when extended to k-mers (where k > 2) is:

Original: [Equation]
Revised: [Equation]

Original: Here, seg ( w 1 . . .
Revised: Here, seg ( w 1 . . .

Original: w n ) is the set of all contiguous segmentations of the n -gram “ w 1 . . .
Revised: w n ) is the set of all contiguous segmentations of the n -gram “ w 1 . . .

Original: w n ” (excluding the identity segmentation).
Revised: w n ” (excluding the identity segmentation).

Original: In a valid segmentation ( σ ), the original sequence “ w 1 . . .
Revised: In a valid segmentation ( σ ), the original sequence “ w 1 . . .

Original: w n ” can be divided into any number of partitions of positive ( > 0 ) size.
Revised: w n ” can be divided into any number of partitions of positive ( > 0 ) size.

Original: For example, say for n = 6 , some of the possible valid segmentations are: “( w 1 . . .
Revised: For example, say for n = 6 , some of the possible valid segmentations are: “( w 1 . . .

Original: w 3 ), ( w 4 . . .
Revised: w 3 ), ( w 4 . . .

Original: w 6 )” or “( w 1 , w 2 ) ( w 3 ), ( w 4 . . .
Revised: w 6 )” or “( w 1 , w 2 ) ( w 3 ), ( w 4 . . .

Original: w 6 )”.
Revised: w 6 )”.

Original: The PMI n formulation many times favors tokens with lower frequency  that is, the number of times the n-gram gene sequence appears in the Human Reference Genome .
Revised: The PMI k formulation many times favors tokens with lower frequency , that is, the number of times the n-gram gene sequence appears in the Human Reference Genome .

Original: We thus impose a discounting factor that penalizes rare tokens (Pantel & Lin, 2002).
Revised: We thus impose a discounting factor that penalizes rare tokens (Pantel & Lin, 2002).

Original: We refer to it as the Normalized PMI n formula, which we ﬁnally use for scoring all the individual n-gram sequences.
Revised: We refer to it as the Normalized PMI k formula, which we finally use for scoring all the individual n-gram sequences.

Original: [Equation]
Revised: [Equation]

Original: Here, f ( w 1 . . .
Revised: Here, f ( w 1 . . .

Original: w n ) refers to the frequency of occurrence of the n-gram sequence of w 1 . . .
Revised: w k ) refers to the frequency of occurrence of the k-mer sequence of w 1 . . .

Original: w n .
Revised: w k .

Original: c refers to the minimum frequency of occurrence ( a constant value used as a threshold to remove rare tokens ).
Revised: c refers to the minimum frequency of occurrence ( a constant value used as a threshold to remove rare tokens ).

Original: We determine the threshold c in Equation 2 based on the frequency distributions of the entire collection of n-gram sequences (around 199 Million).
Revised: We determine the threshold c in Equation 2 based on the frequency distributions of the entire collection of k-mers (around 199 Million).

Original: In our case, we choose c equal to 101 which puts a cut-off beyond the ﬁrst quartile ( 25 percentile) of n-gram frequencies.
Revised: In our case, we choose c equal to 101 which puts a cut-off beyond the first quartile ( 25 percentile) of k-mer frequencies.

Original: In this paper, we focus only on computing Normalized-PMI n for all k-mer sequences where k =6 , and use a top-down memorization approach to reduce algorithmic complexity, by storing already computed PMI scores in memory.
Revised: In this paper, we focus only on computing Normalized-PMI n for all k-mer sequences where k =6 , and use a top-down memorization approach to reduce algorithmic complexity, by storing already computed PMI scores in memory.

Original: As the ﬁnal step, we develop a ranked list ( RANK ) of all 6-mers ( 4096 in total) based on the decreasing order of Normalized-PMI n .
Revised: As the final step, we develop a ranked list ( RANK ) of all 6-mers ( 4096 in total) based on the decreasing order of Normalized-PMI k .

Original: Next, we discuss how we use the
Revised: Next, we discuss how we use the

Original: PMI scores as a measure to choose tokens to be masked during MLM training.
Revised: PMI scores as a measure to choose tokens to be masked during MLM training.

Original: 4. PMI- MASKING S TRATEGY
Revised: 4. PMI- MASKING S TRATEGY

Original: The strategy aims to mask all the nucleotides simultaneously in the most correlated spans.
Revised: The strategy aims to mask all the nucleotides simultaneously in the most correlated spans.

Original: It minimizes information leakage and helps the system to learn deeper patterns.
Revised: It minimizes information leakage and helps the system to learn deeper patterns.

Original: Side by side, we would like to preserve the beneﬁt drawn out of the traditional random masking strategy.
Revised: Side by side, we would like to preserve the benefit drawn out of the traditional random masking strategy.

Original: Thus, to choose the tokens for MLM training, we perform the following steps.
Revised: Thus, to choose the tokens for MLM training, we perform the following steps.

Original: Randomly select m nucleotides as mask center spread uniformly over the DNA string.
Revised: Randomly select m nucleotides as mask center spread uniformly over the DNA string.

Original: Corresponding to a selected nucleotide (say DNA[ i ]), obtain a PMI score MPMI i in a folllowing manner.
Revised: Corresponding to a selected nucleotide (say DNA[ i ]), obtain a PMI score MPMI i in a following manner.

Original: (a) A 6-mer token corresponding to DNA[ i ] : T 6 [ i ] =
Revised: (a) A 6-mer token corresponding to DNA[ i ] : T 6 [ i ] =

Original: { DNA[ i -2] · · · DNA[ i +3] } (b) Obtain the PMI rank of the token as PMI-Rank(T 

Revised: { DNA[ i -2] · · · DNA[ i +3] } (b) Obtain the PMI rank of the token as PMI-Rank(T

Original: [ i ])
Revised: [ i ])

Original: (c) MPMI i = max i
Revised: (c) MPMI i = max i

Original: +3 i
Revised: +3 i

Original: = i − 2 (PMI-Rank(T 

Revised: = i − 2 (PMI-Rank(T

Original: [ i ]) 3.
Revised: [ i ]) 3.

Original: Divide the m nucleotides into two sets based upon their MPMI scores, where the high setting is the m/ 2 nucleotides with the highest MPMI scores.
Revised: Divide the m nucleotides into two sets based upon their MPMI scores, where the high setting is the m/ 2 nucleotides with the highest MPMI scores.

Original: For the high set, take all the nucleotides in T[ i ] (this ensures  masking of correlated spans together).
Revised: For the high set, take all the nucleotides in T[ i ] (this ensures the masking of correlated spans together).

Original: In contrast, for the low set, take only the corresponding nucleotide DNA
Revised: In contrast, for the low set, take only the corresponding nucleotide DNA

Original: [ i ] (this mimics a random masking strategy) for masking.
Revised: [ i ] (this mimics a random masking strategy) for masking.

Original: Determining the value of m : To mask a six bp length sequence that is all the nucleotides in a particular token T 6 [ i ], we need to mask a span of contiguous 11 tokens (6 mask centers, two tokens to the left and three tokens right) while as mentioned a single nucleotide induce masking of 6 tokens.
Revised: Determining the value of m : To mask a six bp length sequence that is all the nucleotides in a particular token T 6 [ i ], we need to mask a span of contiguous 11 tokens (6 mask centers, two tokens to the left and three tokens right) while as mentioned a single nucleotide induce masking of 6 tokens.

Original: Thus, the expected mask span length per mask center is computed as: 0 .
Revised: Thus, the expected mask span length per mask center is computed as: 0 .

Original: 5 ∗ 6 + 0 .
Revised: 5 ∗ 6 + 0 .

Original: 5 ∗ 11 = 8 . 5 .
Revised: 5 ∗ 11 = 8 . 5 .

Original: Followingly, the mlm probability is updated from 2 .
Revised: Followingly, the mlm probability is updated from 2 .

Original: 5% to 1 .
Revised: 5% to 1 .

Original: 765% (= 15% / 8 . 5 ).
Revised: 765% (= 15% / 8 . 5 ).

Original: Hence if the DNA length is 512, m ≈ 9.
Revised: Hence if the DNA length is 512, m ≈ 9.

Original:  E XPERIMENTAL S ETUP
Revised:  E XPERIMENTAL S ETUP

Original: Here, we ﬁrst provide the dataset details and evaluation setup, followed by the model training and baseline model details.
Revised: Here, we first provide the dataset details and evaluation setup, followed by the model training and baseline model details.

Original: 5. D ATASETS
Revised: 5. D ATASETS

Original: We use three benchmark datasets of gene sequence classiﬁcation for evaluation purposes.
Revised: We use three benchmark datasets of gene sequence classification for evaluation purposes.

Original: The two datasets of promoter region prediction are not directly made available and involve a signiﬁcant amount of effort (including a paper implementation) for its construction.
Revised: The two datasets of promoter region prediction are not directly made available and involve a significant amount of effort (including a paper implementation) for their construction.

Original: The enhancer cohn prediction dataset is directly made available from prior works (Martinek et al., 2022).
Revised: The enhancer cohn prediction dataset is directly made available from prior works (Martinek et al., 2022).

Original: Promoter Region Prediction (Prom-core and Prom-300).
Revised: Promoter Region Prediction (Prom-core and Prom-300).

Original: A promoter is a DNA region typically located upstream of the gene, which is the site of transcription initiation.
Revised: A promoter is a DNA region typically located upstream of the gene, which is the site of transcription initiation (as defined in Zaheer et al.

Original: The task is to classify a given DNA fragment as a promoter or non-promoter sequence, as followed by previous studies (Ji et al., 2021;
Revised: The task is to classify a given DNA fragment as a promoter or non-promoter sequence, as followed by previous studies (Ji et al., 2021;

Original: Zaheer et al., 2020).
Revised: Zaheer et al., 2020).

Original: However, we follow the instructions of Oubounyt et al.
Revised: However, we follow the instructions of Oubounyt et al.

Original: (2019) including the negative data creation) since the datasets are not provided by the authors.
Revised: (2019) including the negative data creation) since the datasets are not provided by the authors.

Original: Thus, we obtained human TATA and non-TATA promoter data, i.e., including promoter sequences with and without a TATA box (a common promoter-related motif found between − 30 to − 25 bp (upstream) of a gene’s transcription start site), from the Eukaryotic Promoter Database (Dreos et al., 2012), using the website API of the EPD selection tool (EPD, 2022).We extracted − 249 to +50 bp sequences around TSS for the Prom-300 setting and − 34 to +35 bp for Prom-core setting.
Revised: Thus, we obtained human TATA and non-TATA promoter data, i.e., including promoter sequences with and without a TATA box (a common promoter-related motif found between − 30 to − 25 bp (upstream) of a gene’s transcription start site), from the Eukaryotic Promoter Database (Dreos et al., 2012), using the website API of the EPD selection tool (EPD, 2022).We extracted − 249 to +50 bp sequences around TSS for the Prom-300 setting and − 34 to +35 bp for Prom-core setting.

Original: We perform the standard train-test split of 70% and 30% , which leads to 53276 and 5920 data points respectively.
Revised: We perform the standard train-test split of 70% and 30% , which leads to 53276 and 5920 data points respectively.

Original: Enhancer Cohn Prediction (Cohn-enh).
Revised: Enhancer Cohn Prediction (Cohn-enh).

Original: An enhancer is a DNA sequence that can bind speciﬁc proteins and increase the chance of transcription of a particular gene.
Revised: An enhancer is a DNA sequence that can bind specific proteins and increase the chance of transcription of a particular gene.

Original: Unlike promoters, enhancers do not need to be close to TSS (it might be several milion basepairs away).

Original: This dataset has been adapted from Cohn et al.
Revised: This dataset has been adapted from Cohn et al.

Original: and is made available as a benchmark dataset by Martinek et al.
Revised: and is made available as a benchmark dataset by Martinek et al.

Original: (2022) in their Github repository (Gresova et al., 2022).
Revised: in their Github repository (Gresova et al., 2022).

Original: Here, the input is a DNA sequence ofbp in length and a binary classiﬁcation task.
Revised: Here, the input is a DNA sequence of 500 bp in length and a binary classification task.

Original: We use the train-test split used by Martinek et al.
Revised: We use the train-test split used by Martinek et al.

Original: (2022) that leads to 20843 and 6948 data points as train and test datasets respectively.
Revised: (2022) that leads to 20843 and 6948 data points as train and test datasets respectively.

Original: 5. E VALUATION S ETUP
Revised: 5. E VALUATION S ETUP

Original: We report the standard metrics of accuracy  and AUC (stands for Area Under the Receiver Operating Characteristic Curve) used for classiﬁcation tasks where the class labels are balanced.
Revised: We report the standard metrics of accuracy (used for performance comparison) and AUC (stands for Area Under the Receiver Operating Characteristic Curve) used for classification tasks where the class labels are balanced.

Original: All reported values are rounded off to three decimal places, and we use accuracy as the metric for performance comparison.

Original: We follow the standard evaluation setup used in the few-shot text classiﬁcation setting (Schick & Sch¨utze, 2021a;b).
Revised: We follow the standard evaluation setup used in the few-shot text classification setting (Schick & Sch¨utze, 2021a;b).

Original: Thus, we assume that we do not have access to a validation dataset (development set) to optimize the hyperparameters and investigate the performance for different training set sizes where t = 10 , 50 , 100 , 500 , and 1000 .
Revised: Thus, we assume that we do not have access to a validation dataset (development set) to optimize the hyperparameters and investigate the performance for different training set sizes where t = 10 , 50 , 100 , 500 , and 1000 .

Original: For each t , we obtain the training set τ by randomly choosing t number of examples from each class.
Revised: For each t , we obtain the training set τ by randomly choosing t number of examples from each class.

Revised: We report the mean and standard deviation of accuracy and AUC by running the experiments ten times by randomly choosing t different training (fine-tuning) examples per class as well as a random seed in each run.

Original: 5. I MPLEMENTATION D ETAILS
Revised: 5. I MPLEMENTATION D ETAILS

Original: Pretraining.
Revised: Pretraining.

Original: The authors of DNABert train for 120K steps on 8 NVIDIA 2080Ti GPUs, which takes 25 days to complete.
Revised: The authors of DNABert train for 120K steps on 8 NVIDIA 2080Ti GPUs, which takes 25 days to complete.

Original: LOGO is trained for 25 epochs which took around 74 days to complete using four Nvidia Tesla V100 32G GPUs.
Revised: LOGO is trained for 25 epochs which took around 74 days to complete using four Nvidia Tesla V100 32G GPUs.

Original: In this paper, we train all the pretrained models and their variants for only 10000 steps, which takes about 2 .
Revised: In this paper, we train all the pretrained models and their variants for only 10000 steps, which takes about 2 .

Original: 5 days to complete for DNABert and around 20 hours for LOGO using four GTX 1080Ti 11GB.
Revised: 5 days to complete for DNABert and around 20 hours for LOGO using four GTX 1080Ti 11GB.

Original: We select such a setup for two reasons — (i) to explore different pretrained model variants in a reasonable time because we observe that the perplexity of DNABert (SOTA model) has coverged to a low score of 2 . 526 and is stable over the last 3000 pretraining steps (see Figure 1).
Revised: We select such a setup for two reasons — (i) to explore different pretrained model variants in a reasonable time because we observe that the perplexity of DNABert (SOTA model) has converged to a low score of 2 . 526 and is stable over the last 3000 pretraining steps (see Figure 1).

Original: (ii) As observed by Levine et al. (2020), PMI-masking learns fast and is thus quite efﬁcient to reapin the beneﬁt even with  lower number of pretraining steps.
Revised: (ii) As observed by Levine et al. (2021), PMI-masking learns fast and is thus quite efficient to reap the benefit even with a lower number of pretraining steps.

Original: We use the same hyperparameter setting as the original SOTA modelsuch as the learning rate of 4 e − 4 , train batch size per GPU as 10 and number of gradient accumulation steps as 25 (see Section A.3 to know more about training details and hyperparameters used).
Revised: We use the same hyperparameter setting as the original SOTA model (see Section A.2 to know more about training details and hyperparameters used).

Original: Finetuning for gene sequence modeling downstream tasks.
Revised: Finetuning for gene sequence modeling downstream tasks.

Original: We use the same model ﬁnetuning hyperparameters and training setup as used by DNABert for the 500 and 1000-shot setting (Ji et al., 2021).
Revised: We use the same model finetuning hyperparameters and training setup as used by DNABert for the 500 and 1000-shot setting (Ji et al., 2021).

Original: The model is ﬁnetuned for 5 epochs at a learning rate of 5 e − 5 , warmup steps percentage of 10% , hidden dropout probability of 0 . 1 , and uses the AdamW optimizer.
Revised: The model is finetuned for 5 epochs at a learning rate of 5 e − 5 , warmup steps percentage of 10% , hidden dropout probability of 0 . 1 , and uses the AdamW optimizer.

Original: We only update the hyperparameters in order to adapt to the low few-shot setting (10, 50 and 100-shot), where we increase the learning from 5 e − 5 to 4 e − 4 and reduce the per gpu train batch size from 15 to 5 .
Revised: We only update the hyperparameters in order to adapt to the low few-shot setting (10, 50, and 100-shot), where we increase the learning from 5 e − 5 to 4 e − 4 and reduce the per GPU train batch size from 15 to 5 .

Original: The idea behind these two hyperparameter changes is to indirectly increase the number of weight update operations during ﬁnetuning, without overﬁtting (given the limited training dataset size).
Revised: The idea behind these two hyperparameter changes is to indirectly increase the number of weight update operations during finetuning, without overfitting (given the limited training dataset size).

Original: We also increase the number of training epochs from 5 to 20 to mitigate ﬁnetuning performance stability issues that may occur due to random initialization in low-resource settings, as observed by Mosbach et al.
Revised: We also increase the number of training epochs from 5 to 20 to mitigate finetuning performance stability issues that may occur due to random initialization in low-resource settings (Mosbach et al., 2021).

Original: 5. B ASELINE M ODELS
Revised: 5. B ASELINE M ODELS

Original: We evaluate our work on two gene transformer-based models - DNABert (Ji et al., 2021) and
Revised: We evaluate our work on two gene transformer-based models - DNABert (Ji et al., 2021) and

Original: LOGO (Yang et al., 2021).
Revised: LOGO (Yang et al., 2021).

Original: Both of these models follow random masking during the MLM training step instead of the proposed PMI-masking strategy.
Revised: Both of these models follow random masking during the MLM training step instead of the proposed PMI-masking strategy.

Original: Weprovide a detailed description of the DNABert and LOGO models in Section 3. We will refer to the SOTA models that use random masking as the original SOTA model (ORI) without any PMI-guided masking. 
Revised: We will refer to the SOTA models that use random masking as the original SOTA model (ORI) without any PMI-guided masking. We use the original DNABert model pretrained on 120K steps based on the pretrained model weights provided by Ji et al.

Revised: (2021), as a baseline model and denote it as ORI 120K model.

Original: Fixed PMI-guided Masking Vocabulary (PMI-VOCAB) :
Revised: Fixed PMI-guided Masking Vocabulary (PMI-VOCAB) :

Original: Similarly to the creation of a PMImasking vocabulary by Levine et al.
Revised: Similarly to the creation of a PMImasking vocabulary by Levine et al.

Original: (2020), we create a masking vocabulary
Revised: (2021), we create a masking vocabulary

Original: ≈ 10 times the
Revised: ≈ 10 times the

Original: DNABert vocabulary size of 4101 tokens.
Revised: DNABert vocabulary size of 4101 tokens.

Original: We ﬁrst select all possible k-mer sequences ( 2 ≤ k ≤ 10 ) whose frequency of occurrence is ≥ 10000 .
Revised: We first select all possible k-mer sequences ( 2 ≤ k ≤ 10 ) whose frequency of occurrence is ≥ 10000 .

Original: We then rank them using our proposed PMI metric and select the top 40000 as the masking vocabulary.
Revised: We then rank them using our proposed PMI metric and select the top 40000 as the masking vocabulary.

Original: We explore multiple hyperparameter settings formodel pretraining (HGA and WC) and one for ﬁnetuning (FS).
Revised: We explore multiple hyperparameter settings for pretraining (HGA and WC) and finetuning (FS).

Original: Half Gradient Accumulation (HGA) :
Revised: Half Gradient Accumulation (HGA) :

Original: The gradient accumulation steps parameter is halved, reducing it from 25 (default DNABert conﬁguration) to 12 steps and consequently reduces the effective batch size (it is the product of per GPU train batch size, GPU count and gradient accumulation steps) by 50% .
Revised: The gradient accumulation steps parameter is halved, reducing it from 25 (default DNABert configuration) to 12 steps and consequently reduces the effective batch size (it is the product of per GPU train batch size, GPU count and gradient accumulation steps) by 50% .

Original: This aims to reduce the generalization gap issue that arises when the training batch size is too large (Keskar et al., 2017; Hoffer et al., 2017).
Revised: This aims to reduce the generalization gap issue that arises when the training batch size is too large (Keskar et al., 2017; Hoffer et al., 2017).

Original: Warmup Correction (WC) :
Revised: Warmup Correction (WC) :

Original: Instead of the ﬁxed number of 10000 warmup steps as in the default conﬁguration of DNABert, we compute it as a percentage of the maximum number of pretraining steps.
Revised: Instead of the fixed number of 10000 warmup steps as in the default configuration of DNABert, we compute it as a percentage of the maximum number of pretraining steps.

Original: In the original conﬁguration, only the ﬁrst 10K steps out of 200K ( 5% of the maximum number of steps).
Revised: In the original configuration, only the first 10K steps out of 200K ( 5% of the maximum number of steps).

Original: Since we reduce the maximum steps limit from 200K to 10K, we adjust the warmup number as 500 ( 5% of 10000 steps) accordingly.
Revised: Since we reduce the maximum steps limit from 200K to 10K, we adjust the warmup number as 500 ( 5% of 10000 steps) accordingly.

Original: The importance of warmup in optimization is highlighted in multiple recent studies (Xiong et al., 2020; Liu et al., 2020; Mosbach et al., 2021).
Revised: The importance of warmup in optimization is highlighted in multiple recent studies (Xiong et al., 2020; Liu et al., 2020; Mosbach et al., 2021).

Original: Finetuning Stability (FS) :
Revised: Finetuning Stability (FS) :

Original: Longer ﬁnetuning epochs of 20 instead of the original conﬁguration of 5 epochs, along with higher learning rate from 5 e − 5 to 4 e − 4 .
Revised: Longer finetuning epochs of 20 instead of the standard 5 epochs, along with higher learning rate from 5 e − 5 to 4 e − 4 .

Original: This is done to improve the ﬁnetuning stability in model performance and alleviates issues due to random initialization  in low-resource settings (Mosbach et al., 2021).
Revised: It improves the finetuning stability in model performance and solves random initialization issues in low-resource settings (Mosbach et al., 2021).

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: Here, we investigate two research questions: (i) Does PMI-masking strategy lead to reasonable performance improvement over random masking in different few-shot settings?
Revised: Here, we investigate two research questions: (i) Does PMI-masking strategy lead to reasonable performance improvement over random masking in different few-shot settings?

Original: (ii) Does the PMIbased ranking show association with conserved DNA sequence motifs?
Revised: (ii) Does the PMIbased ranking show association with conserved DNA sequence motifs?

Original: 6. PMI- MASKING R ESULTS
Revised: 6. PMI- MASKING R ESULTS

Original: We present the results of the model performance comparison in Table 1.
Revised: We present the results of the model performance comparison in Table 1.

Original:  The  results  are reported based on a single run with a ﬁxed seed value of 42 .
Revised: PMI-best represents the best performance of the two model variants: PMI (WC + FS) and PMI (WC + HGA + FS); The performance results for the hyperparameter-based model variants are provided in Table 2. In the case of DNABert , PMI-best outperforms all the baseline models for the task of Prom-300 and Cohn-enh, except for 1000-shot Prom-300 setting, which also includes the original DNABert model trained for 120K steps (ORI 120K model).

Original: We observe that at least one PMI-maskingbased modelconsistently outperformsthe original SOTA model (ORI) in every few-shot setting and for both the SOTA models. PMI-best represents the best performance of the two model variants:
Revised: ORI 120K model outperforms PMI-best on shallow data settings (10, 50, and 100-shot) of the Prom-core task, whereas PMI-best outperforms again in 500 and 1000shot settings. This may be because Prom-core is an easier task with a much shorter context (70 bp.

Original: PMI (WC + FS) and PMI (WC + HGA + FS); The performance results for the hyperparameterbased model variants are provided in Table 2.
Revised: in length as compared to 300 and 500 bp in length in the case of Prom-300 and Cohn-enh task).

Original: In the case of DNABert , PMI-best achieves the best performance for almost all the few-shot settings over the three benchmark tasks, indicating that the proposed PMI-masking strategy teaches valuable information to the model during pretraining.
Revised: Thus, the ORI 120K model may simply memorize the gene sequence patterns instead of actually learning intrinsic (or extra) knowledge of gene sequences. Therefore, we also observe that such effect of memorization vanishes is higher data settings (above 500-shot).

Original: We observe the highest performance improvement of PMI-best over ORI of 10 . 12% and 12 .43% in
Revised: We observe the highest performance improvement of PMI-best over ORI of 7 . 47% and 4 .

Original:  DNABert and LOGO, respectively, in the 10-shot (very low data) setting.
Revised: 68% in DNABert and LOGO, respectively, in the 10-shot (shallow data) setting.

Original: However, the performance gap slowly reduces in higher data settings  ( 2 . 39% , 3 .
Revised: However, the performance gap slowly reduces in higher data settings in case of DNABert ( 3 . 47% , 0 .

Original: 19% , 1 . 29% and 1 . 28% for 50, 100, 500, and 1000-shot, respectively).
Revised: 13% and 0% for 50, 100, 500, and 1000-shot, respectively).

Original: PMI-VOCAB model outperforms PMI-best in two cases of Prom-core 10-shot and Cohn-enh 50-shot setting.

Original: This may be because PMI-VOCAB is not restricted to only 6-mer PMI tokens and accommodates more extended PMI tokens up to 10-mers.

Original: In the case of LOGO , both for random masking and PMI, we consider the best performing model setting of DNABert corresponding to a particular x -shot.
Revised: In the case of LOGO , both for random masking and PMI, we consider the best-performing model setting of DNABert corresponding to a particular x -shot.

Original: We observe that the PMI-masking-guided model outperforms random masking-based SOTA models (ORI) in all settings across the three benchmark datasets, particularly by a large margin in shallow data settings (10 and 50-shot) of Prom-core and Prom-300.
Revised: We observe that the PMI-masking-guided model outperforms random masking-based SOTA models (ORI) in all settings across the three benchmark datasets except for 500-shot Prom core setting.

Original: The model accuracy improves by 14 . 82% and 21 .46% for Prom-core and
Revised: It particularly improves by a large margin in shallow data settings of Prom-core (accuracy improvement of 6 . 72% and 8 .

Original: Prom-300, respectively, in the 10-shot setting. Similarly, in the 50-shot setting, the performance increases by 9 . 27% and 4 . 
Revised: 14% in 10 and 50-shot respectively) and Prom-300 ( 8 . 09% in 50-shot).

Original: We next report the performance improvement between PMI-masking and random masking (ORI model variant) in DNABert versus LOGO in terms of average accuracy over all few-shot settings for each task: Prom-core = 1 .
Revised: We next report the performance improvement between PMI-masking and random masking (ORI model variant) in DNABert versus LOGO in terms of average accuracy over all few-shot settings for each task: Prom-core = 1 .

Original: 58% vs. 6 . 25% , Prom-300 = 5 .
Revised: 59% vs. 3 . 72% , Prom-300 = 2 .

Original: 78% vs. 6 .
Revised: 61% vs. 7 .

Original: 73% , and Cohn-enh = 2 .
Revised: 67% , and Cohn-enh = 2 .

Original: 41% vs. 2 . 89% .
Revised: 13% vs. 1 .

Original: To keep the improvement results in perspective, we assess the task difﬁculty of each dataset based on its performance difference between very low (10-shot) and relatively higher (1000-shot) data settings.

Original: We observe that the addition of more ﬁnetuning data shows the lowest performance improvement  for Cohn-enh ( 8 . 94% and 9 .
Revised: We observe that the addition of more finetuning data shows the lowest performance improvement (in between 10-shot and 1000-shot) for Cohn-enh ( 13 .

Original: 67% ) as compared to Prom-300 ( 35 .
Revised: 50% and 19 . as compared to Prom-300 ( 32 .

Original: 98% and 43 .
Revised: 84% and 64 .

Original: 17% ) and Prom-core ( 22 .
Revised: 74% ) and Prom-core ( 27 .

Original: 63% and 25 . 82% ) tasks in case of DNABert and LOGO respectively.
Revised: 24% and 33 . 89% ) tasks in case of DNABert and LOGO respectively.

Original: However, we observe the effect of PMI-masking on model performance to be higher in the case of 
Revised: However, we observe the effect of PMI-masking on model performance to be higher in the case of LOGO for Prom-core and Prom-300 .

Original: LOGO in comparison to DNABert . It indicates that PMI-masking is more useful for lightweight models like LOGO; heavier models like DNABert might automatically learn a certain amount of span correlations (like PMI) information, thus reducing the independent impact of PMI-masking.
Revised: It indicates that PMI-masking is more beneficial for lightweight models like LOGO; heavier models like DNABert might automatically learn a certain amount of span correlations (like PMI) information, thus reducing the independent impact of PMI-masking.

Original: 6. A BLATION A NALYSIS
Revised: 6. A BLATION A NALYSIS

Original: We show the performance comparison among the different model hyperparameter-based variations (described in Section 5.4) in Table 2.
Revised: We show the performance comparison among the different model hyperparameter-based variations (described in Section 5.4) in Table 2.

Original: We observe that ﬁnetuning stability (FS) works very well with limited training data (10, 50, and 100-shot settings), where substantial tremendous improvement is noticed.
Revised: We observe that finetuning stability (FS) works very well with limited training data (10, 50, and 100-shot settings), where substantial tremendous improvement is noticed.

Original: However, with more training data, such as 500 and 1000-shot settings, we observe a highperformance drop due to overﬁtting.
Revised: However, with more training data, such as 500 and 1000-shot settings, we observe a highperformance drop due to overfitting.

Original: We therefore do not use FS in 500 and 1000-shot settings.
Revised: We, therefore, do not use FS in 500 and 1000-shot settings.

Original: We observe that half gradient accumulation (HGA) helps to improve the model performance in 10 and 50-shot settings (10-shot for Prom-300 andCohn-enh and 50-shot for Prom-core).
Revised: We observe that half gradient accumulation (HGA) helps to improve the model performance in 10 and 50-shot settings (10 and 50-shot for Prom-300 and 50-shot for Cohn-enh).

Original: It is empirically observed that if a large batch size is used to train deep neural networks, the trained models appear to generalize poorly (Keskar et al., 2017; Hoffer et al., 2017).
Revised: It is empirically observed that if a large batch size is used to train deep neural networks, the trained models appear to generalize poorly (Keskar et al., 2017; Hoffer et al., 2017).

Original: HGA reduces the effective batch size by 50% and mitigates the generalization gap issue in such low data settings (10 and 50-shot).
Revised: HGA reduces the effective batch size by 50% and mitigates the generalization gap issue in such low data settings (10 and 50-shot).

Original: However, beyond the 100-shot setting, the impact is either negative or marginal.
Revised: However, beyond the 100-shot setting, the impact is either negative or marginal.

Original: We observe that warmup correction (WC) provides a decent performance improvement across all the few-shot settings ( 12 . 44% in 50-shot, 8 . 58% in 500-shot, etc.)in comparison to ﬁnetuning [Table]
Revised: We observe that warmup correction (WC) provides a decent performance improvement across all the few-shot settings ( 4 . 46% in 50-shot, 1 . 56% in 500-shot, etc.)

Original: steps.

Original: ORI refers to the SOTA model with random masking, and PMI-best refers to the best performance of two PMI-masking model variants: PMI (WC + FS) , PMI (WC + HGA + FS) . FS is not performed where * is marked and for 500 and 1000-shot setting due to model overﬁtting.
Revised: in comparison to finetuning stability (FS) and half gradient accumulation (HGA), where their utility is only observed to lower data settings of 10, 50, and 100-shot. The number of warmup steps in standard pretraining setups is usually

Original: [Table]
Revised: [Table]

Original: since we use mini-batch Gradient Descent, the model noisily converges towards minima and may oscillate far away from the actual minima. As a result, the model convergence is signiﬁcantly delayed, negatively impacting both model optimization and generalization.
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: 6. M OTIF A NALYSIS
Revised: 6. M OTIF A NALYSIS

Original: Motifs are repetitive units having  certain biological signiﬁcance.
Revised: Motifs are repetitive units having a certain biological significance.

Original: We here check whether the correlated tokens identiﬁed by PMI are in fact (part of) such units.
Revised: We here check whether the correlated tokens identified by PMI are in fact (part of) such units.

Original: Association between the top-20 ranked 6-mers based on Normalized-PMI n score and conserved DNA sequence motifs.
Revised: Association between the top-20 ranked 6-mers based on Normalized-PMI n score and conserved DNA sequence motifs.

Original: We analyze whether highly ranked PMI tokens resemble meaningful concepts by checking their overlap with known motifs.
Revised: We analyze whether highly ranked PMI tokens resemble meaningful concepts by checking their overlap with known motifs.

Original: We performed a Google search with the following query template: “AATCTC” dna sequence motif .
Revised: We performed a Google search with the following query template: “AATCTC” DNA sequence motif .

Original: Double quotes are used as a Google wildcard to indicate that the term AATCTC must always be present in the returned results.
Revised: Double quotes are used as a Google wildcard to indicate that the term AATCTC must always be present in the returned results.

Original: We only considered the ﬁrst page of Google results to consider hits to the query.
Revised: We only considered the first page of Google results to consider hits to the query.

Original: Among the top 20 ranked 6-mers, we observed that all except 2 (CCAGGC - rank 9, GCCTGG - rank 10) are previously mentioned in the published biomedical literature.
Revised: Among the top 20 ranked 6-mers, we observed that all except 2 (CCAGGC - rank 9, GCCTGG - rank 10) are previously mentioned in the published biomedical literature.

Original: PMI-based rankings capture motifs present in ﬁnetuning datasets.
Revised: PMI-based rankings capture motifs present in finetuning datasets.

Original: We use the R package rGADEM (Droit et al., 2022) to perform de novo motif discovery.
Revised: We use the R package rGADEM (Droit et al., 2022) to perform de novo motif discovery.

Original: As motif discovery is computationally expensive, we only provide a subset of the data: randomly sampled 1000 (prom300 - 300 bp and enhancers-cohn 500 bp) and 2000 (prom-core - 70 bp) data points.
Revised: As motif discovery is computationally expensive, we only provide a subset of the data: randomly sampled 1000 (prom300 - 300 bp and enhancers-cohn 500 bp) and 2000 (prom-core - 70 bp) data points.

Original: We obtain a total of 12 motifs from the Prom-core and Cohn-enh datasets, which are of varying lengths, but are mostly concentrated around length 5, 6, and 7.
Revised: We obtain a total of 12 motifs from the Prom-core and Cohn-enh datasets, which are of varying lengths but are mostly concentrated around lengths 5, 6, and 7.

Original: We present our best matches and their corresponding ranks in the 6-mer PMI ranked list ( RANK ) in Table 3 (see Table 6 for the complete list, as well as their corresponding consensus logos in Figures 2 and 3).
Revised: We present our best matches and their corresponding ranks in the 6-mer PMI ranked list ( RANK ) in Table 3 (see Table 6 for the complete list, as well as their corresponding consensus logos in Figures 3 and 4).

Original: We observe that most of the 6-mers that match the discovered motifs are ranked very high.
Revised: We observe that most of the 6-mers that match the discovered motifs are ranked very high.

Original: within the top 25 percentile of RANK , that is, a rank of 1024 .
Revised: Within the top 25 percentile of RANK , that is, a rank of 1024 .

Original: We further observe that our top-1 ranked 6-mer is present in motifs of both enhancers and promoters based on de-novo motif discovery and is also mentioned in previous biomedical literature (Chow et al., 1991).
Revised: We further observe that our top-1 ranked 6-mer is present in motifs of both enhancers and promoters based on de-novo motif discovery and is also mentioned in previous biomedical literature (Chow et al., 1991).

Original: The TATA box, a well-known motif for promoters (rows 3 of Table 3) is ranked at 242 ; the best TATA box motif is TATATA , which has a PMI rank of 15 .
Revised: The TATA box, a well-known motif for promoters (rows 3 of Table 3) is ranked at 242 ; the best TATA box motif is TATATA , which has a PMI rank of 15 .

Original: We thus conclude that top-ranked 6-mers (top 25 percentile) have a strong correlation with DNA sequence motifs.
Revised: We thus conclude that top-ranked 6-mers (top 25 percentile) have a strong correlation with DNA sequence motifs.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Gene sequence classiﬁcation is a challenging problem and requires a tremendous engineering effort even to develop the experimental setup.
Revised: Gene sequence classification is a challenging problem and requires a tremendous engineering effort even to develop the experimental setup.

Original: The publicly available infrastructure currently in place needs to be improved to facilitate such research.
Revised: The publicly available infrastructure currently in place needs to be improved to facilitate such research.

Original: We plan to open-source the pretraining and ﬁnetuning datasets and associated codes, along with model weights and checkpoints, with the objective of broader adoption by the research community.
Revised: We plan to open-source the pretraining and finetuning datasets and associated codes, along with model weights and checkpoints, with the objective of broader adoption by the research community.

Original: We believe this would be an important contribution to the work besides the critical ﬁnding that the PMI-masking strategy improves over the random masking strategy in all the few-shot settings for two SOTA models - DNABert and LOGO over three gene sequence classiﬁcation datasets.
Revised: We believe this would be an important contribution to the work besides the critical finding that the PMI-masking strategy improves over the random masking strategy in all the few-shot settings for two SOTA models - DNABert and LOGO over three gene sequence classification datasets.

Original: We also performed a detailed motif analysis and observed a strong correlation between top-ranked PMI tokens and conserved DNA sequence motifs, providing us with a biological reason behind the improvement.
Revised: We also performed a detailed motif analysis and observed a strong correlation between top-ranked PMI tokens and conserved DNA sequence motifs, providing us with a biological reason behind the improvement.

Original: Finally, given the often limited dataset sizes in (personalized) medicine, the present framework of entity-aware pretraining leading to improvement in a few-shot setting needs to be explored in full, which will be our immediate future work.
Revised: Finally, given the often limited dataset sizes in (personalized) medicine, the present framework of entity-aware pretraining leading to improvement in a few-shot setting needs to be explored in full, which will be our immediate future work.

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: The gene sequence data used for both pretraining and ﬁnetuning is obtained from publicly available sources and can be obtained directly without signing any explicit data use agreement.
Revised: The gene sequence data used for both pretraining and finetuning is obtained from publicly available sources and can be obtained directly without signing any explicit data use agreement.

Original: The three benchmark datasets of Prom-core, Prom-300 and Cohn-enh are also used in previous studies for the task of gene sequence classiﬁcation (Zaheer et al., 2020;
Revised: The three benchmark datasets of Prom-core, Prom-300, and Cohn-enh are also used in previous studies for the task of gene sequence classification (Zaheer et al., 2020;

Original: Ji et al., 2021; Martinek et al., 2022;
Revised: Ji et al., 2021; Martinek et al., 2022;

Original: Oubounyt et al., 2019).
Revised: Oubounyt et al., 2019).

Original: Our work does not involve patient-level data for the experiments.
Revised: Our work does not involve patient-level data for the experiments.

Original: We do not foresee any negative social impacts of this work, but of course the accumulation of improvements in ML could be misused as it may give more power to nefarious agents.
Revised: We do not foresee any negative social impacts of this work, but of course, the accumulation of improvements in ML could be misused as it may give more power to nefarious agents.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: We develop the Normalized-PMI n metric and also incorporate it into SOTA models as PMI-masking.
Revised: We develop the Normalized-PMI n metric and also incorporate it into SOTA models as PMI-masking.

Original: In terms of motif analysis, we also perform the de novo motif discovery on the downstream task datasets using the rGADEM R package.
Revised: In terms of motif analysis, we also perform the de novo motif discovery on the downstream task datasets using the rGADEM R package.

Original: We will provide the code which includes the pretraining and ﬁnetuning data construction, together with a detailed reproducibility report (ﬁle name: reproducibilty-report-iclr-2023 ) in the supplementary material.
Revised: We will provide the code which includes the pretraining and finetuning data construction, together with a detailed reproducibility report (file name: reproducibilty-report-iclr-2023 ) in the supplementary material.

Original: We will be releasing the codebase and the pretrained models after the review process.
Revised: We will be releasing the codebase and the pretrained models after the review process.

Original: A S UPPLEMENTARY T EXT M ATERIAL
Revised: A S UPPLEMENTARY T EXT M ATERIAL

Original: In this section we provide the supplementary material associated with the paper.
Revised: In this section we provide the supplementary material associated with the paper

Original: A. B ACKGROUND
Revised: A. B ACKGROUND

Original: Importance of understanding gene regulatory code.
Revised: Importance of understanding gene regulatory code.

Original: The long strands of DNA found in the human chromosomes can be classiﬁed into genes and the genes in turn, comprise coding and non-coding parts.
Revised: The long strands of DNA found in the human chromosomes can be classified into genes and the genes, in turn, comprise coding and non-coding parts.

Original: A coding part encapsulates the information required for converting the nucleotide to a protein .
Revised: A coding part encapsulates the information required for converting the nucleotide to a protein .

Original: These proteins are the building blocks of all tissues.
Revised: These proteins are the building blocks of all tissues.

Original: These genes interact with the non-coding regions which perform gene regulation.
Revised: These genes interact with the non-coding regions which perform gene regulation.

Original: Promoters/Enhancers speed up the process of coding, inhibitors slow down the reaction.
Revised: Promoters/Enhancers speed up the process of coding, inhibitors slow down the reaction.

Original: These non-coding genes are called gene regulatory elements (Yue &
Revised: These non-coding genes are called gene regulatory elements (Yue &

Original: Wang, 2018a).
Revised: Wang, 2018a).

Original: The non-coding regions, accounting for over 98% of the whole genome, implement signiﬁcant yet largely unknown regulatory functions.
Revised: The non-coding regions, accounting for over 98% of the whole genome, implement significant yet largely unknown regulatory functions.

Original: Recent large consortia projects, including the
Revised: Recent large consortia projects, including the

Original: ENCyclopedia of DNA Elements (ENCODE)
Revised: ENCyclopedia of DNA Elements (ENCODE)

Original: (Consortium, 2012), Roadmap Epigenomics (Kundaje et al., 2015), and the Genomics of Gene Regulation (GGR), have produced  large amount of experimental mapping readouts to help annotate non-coding genome in speciﬁc tissues or cell-lines.
Revised: (Consortium, 2012), Roadmap Epigenomics (Kundaje et al., 2015), and the Genomics of Gene Regulation (GGR), have produced a large number of experimental mapping readouts to help annotate non-coding genome in specific tissues or cell-lines.

Original: On the other hand, Genome-wide association studies (GWAS) have discovered that  vast majority ( > 90% ) of associated genome loci for complex disease and traits fall in non-coding regions (Visscher et al., 2017).
Revised: On the other hand, Genome-wide association studies (GWAS) have discovered that the vast majority ( > 90% ) of associated genome loci for complex disease and traits fall in non-coding regions (Visscher et al., 2017).

Original: MLM training.
Revised: training.

Original: Yamaguchi et al.
Revised: Yamaguchi et al.

Original: (2021) explore alternative pretraining tasks compared to MLM such as shufﬂed word detection, random word detection, manipulated word detection (Shufﬂe +
Revised: (2021) explore alternative pretraining tasks compared to MLM such as shuffled word detection, random word detection, manipulated word detection (Shuffle +

Original: Random), masked token type classiﬁcation and masked ﬁrst character prediction.
Revised: Random), masked token type classification, and masked first character prediction.

Original: Here, we choose the original DNABert conﬁguration of MLM without Next Sentence Prediction and experiment with multiple masked token selection strategies.
Revised: Here, we choose the original DNABert configuration of MLM without Next Sentence Prediction and experiment with multiple masked token selection strategies.

Original: A.M ETHODOLOGY O VERVIEW OF P ROPOSED PMI M ASKING S TRATEGY A. P ARAMETER COMPARISON OF THE SOTA MODELS
Revised: A. P ARAMETER COMPARISON OF THE SOTA MODELS

Original: [Table]
Revised: [Table]

Revised: In the current setup, the models are fine-tuned on taskspecific data for 5 epochs with warmup percentage as 0 . 1 , hidden dropout probability as 0 . 1 , and weight decay as 0 . 01 .

Original: . A. E XPERIMENTAL R ESULTS
Revised: A. E XPERIMENTAL R ESULTS

Revised: [Table]

Revised: A.3. A NALYZING THE EFFECT OF FUNCTIONAL GENETIC VARIANTS

Revised: We aim to reproduce the variant analysis conducted by Ji et al.

Revised: (2021), using dbSNP (Sherry et al., 2001) and ClinVar (Landrum et al., 2013), to compare the performance of PMI-masking-based model (PMI) with the original SOTA model (ORI) for the task of identifying functional genetic variants (using DNABert as the ORI model).

Revised: 400 , 000 variants are retrieved from dbSNP, and the corresponding genomic sequences (both original and mutated) are constructed by the original authors; this dataset is publicly available.

Revised: When the original and mutated sequences offer significantly different prediction probabilities, the variant is queried in ClinVar to ascertain their importance.

Revised: Since we do not have access to the specific code used by the authors to evaluate the importance of a given variant using Clinvar, we instead use the same finetuned DNABert model, pretrained for 120K steps and finetuned on the 10-shot Prom-core dataset (ORI 120K performs best in Prom-core low-resource setting) and obtain the original weights used by the authors (DNA, 2021)).

Revised: We will look to evaluate settings other than 10-shot for the sake of completeness as a future research direction.

Revised: As a result, DNABERT demonstrated its capability to capture and propose new and significant (disease-specific) variants in the future.

Revised: Experimental Details.

Revised: The differences in prediction probabilities for the dataset mentioned above ( 400 , 000 data points) in the 10-shot setting and the data points are ranked in non-increasing order of the difference value.

Revised: At this point, we obtain an individual ranked list of data points for the ORI and PMI models.

Revised: We use the ranked (constructed similarly to ORI and PMI models) based on the finetuned DNABert model provided by the authors (described in the above paragraph) as ground truth, i.e., a proxy to identify all possible important functional variants.

Revised: The degree of overlap with the ground-truth ranked list for multiple top-N settings for the ORI and PMI setting is provided in

Revised: Figure 2.

Revised: We observe that the PMI-guided DNABert model consistently reports higher overlap over the original model (ORI) in different top-N settings ( 5000 ≤ N ≤ 50000 ).

Revised: Thus, we conclude that

Revised: PMI-masking helps incorporate intrinsic (or relevant) genomics information into ORI models like

Revised: DNABert and LOGO.

Revised: The performance improvement is wider than just gene sequence classification tasks.

Revised: A. M OTIF A NALYSIS

Revised: Dataset

Revised: Motifs

Revised: Normalized PMI rank (

Original: [Figure]
Revised: [Figure]

Revised: 5 for 5-mers) (out of 4096)

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: AGGAGG (rank 56), GAGGAG (rank 278)

Revised: [Figure]

Revised: [Figure]

Revised: CTGGG

Original: [Table]
Revised: [Figure] [Figure] CCTGGCC [Figure]

Revised: CCTGGC (rank 25), CTGGCC (rank 129)

Revised: Cohn

Revised: [Figure]

Revised: enh yy CCAG

Revised: [Figure]

Revised: n 302, 593, 1247,

Revised: Cohn

Original: [Table]
Revised: [Figure] enh TTTTTT TTTn [Figure]

Revised: sw