igemugm/dnabert-nicotiana

🧬 DNABERT-2 Domain-Adaptive Pretraining (DAP) on Nicotiana tabacum

This model is a domain-adaptively pre-trained DNABERT-2 on Nicotiana tabacum DNA sequences.
The purpose of this pretraining is to adapt the base DNABERT-2 model to capture plant-specific DNA patterns before fine-tuning on downstream tasks.

---

📘 Overview

• Base model: zhihan1996/DNABERT-2-117M
• Pretraining task: Masked Language Modeling (MLM)
• Domain: Nicotiana tabacum DNA sequences
• Tokenizer: DNABERT-2 tokenizer
• Dataset size: ~1% of available Nicotiana sequences
• Objective: Predict 15% randomly masked tokens in each sequence

---

⚙️ Training Details

• Training epochs: 10
• Early stopping: patience = 3 (monitoring validation loss)
• Learning rate: 2e-5
• Weight decay: 0.01
• Batch size: 8 per device
• Sequence length: max 1024 tokens
• Padding: Dynamic, via DataCollatorForLanguageModeling

Training procedure:

1. Tokenize DNA sequences with DNABERT-2 tokenizer
2. Randomly mask 15% of tokens
3. Train with MLM objective
4. Apply early stopping based on validation loss

---

📊 Evaluation

Both training and validation losses gradually decreased, showing that the model learned patterns in Nicotiana tabacum DNA sequences without overfitting.
Final training loss: ~5.55, validation loss: ~5.53. Final loss are relatively high due to the limited dataset (1% of total available data).

---

💡 Usage

This model can be used for masked language modeling tasks or as a pretraining checkpoint for downstream tasks like stress region prediction or other plant DNA sequence analyses.

from transformers import AutoTokenizer, BertModel
import torch

# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained("igemugm/dnabert-nicotiana", trust_remote_code=True)
model = BertModel.from_pretrained("igemugm/dnabert-nicotiana", trust_remote_code=True)

# Input DNA sequence
sequence = "ACGTAGCATCGGATCTATCTATCGACACTTGGTTATCGATCTACGAGCATCTCGTTAGC"
inputs = tokenizer(sequence, return_tensors="pt")

# Forward pass for hidden states
with torch.no_grad():
    outputs = model(**inputs)
    hidden_states = outputs.last_hidden_state  # [batch, seq_len, hidden_dim]

# CLS token embedding
cls_embedding = hidden_states[:, 0, :]  # [batch, hidden_dim]

# Mean pooling embedding
mean_embedding = hidden_states.mean(dim=1)  # [batch, hidden_dim]

print("CLS embedding shape:", cls_embedding.shape)
print("Mean embedding shape:", mean_embedding.shape)