seqSight/seqlens-v2-micro-16k

SeqLens v2 Micro 16K

A compact genomic language model pre-trained on prokaryotic genomes for microbial bioinformatics tasks.

Model Description

SeqLens v2 is a bidirectional genomic language model built on BiMamba (bidirectional Mamba2 SSM) with interleaved sliding-window attention. It is designed for microbial genomics — taxonomic classification, antimicrobial resistance detection, plasmid identification, and metagenomic analysis.

The Micro variant is the smallest in the SeqLens v2 family, targeting high-throughput, low-latency inference.

Property	Value
Parameters	10.3M
Hidden dimension	256
Layers	8 (BiMamba) + 2 (sliding-window attention at layers 3, 7)
Context length	16,384 tokens (single nucleotide)
Vocabulary	A, T, G, C, N, [CLS], [SEP], [PAD], [MASK] (9 tokens)
Pre-training objective	Masked Language Modeling (MLM), 15% mask rate
Architecture	BiMamba2 + chunked sliding-window attention + SwiGLU FFN

Architecture Details

• BiMamba blocks: Bidirectional Mamba2 SSM — processes sequences in both forward and reverse directions using shared weights. Provides O(L) scaling with sequence length.
• Sliding-window attention: Applied every 4th layer with window size 512. Captures fine-grained local patterns (codons, motifs) that SSMs can miss.
• Attention-weighted pooling: Learned pooling for sequence-level embeddings (superior to mean pooling for downstream tasks).
• SwiGLU FFN: Gated feed-forward with 4× expansion at each layer.

Training

Data

Pre-trained on prokaryotic genomes from OpenGenome2 (Apache 2.0):

• GTDB v220: 113,379 species-cluster representative genomes
• Single-nucleotide tokenization, 16,384 bp chunks
• Quality filtered: sequences with >10% N or low Shannon entropy excluded

Hyperparameters

Parameter	Value
Optimizer	AdamW (β₁=0.9, β₂=0.98, ε=1e-8)
Learning rate	1e-3 (cosine decay to 1e-5)
Warmup	500 steps
Weight decay	0.1
Gradient clipping	1.0
Precision	BF16 mixed
Batch size	64 effective (8 × 8 GPUs)
Total steps	10,000
Tokens seen	~1.2B

Compute

Resource	Value
Hardware	8× NVIDIA A100-SXM4-80GB
Training time	57 minutes
Framework	PyTorch 2.6.0 + mamba-ssm 2.2.4

Evaluation

Coding vs Non-coding Classification (linear probe, frozen backbone)

Model	Params	Accuracy	F1
SeqLens v2 Micro	10M	0.911	0.911
SeqLens v1 (89M)	89M	0.687	0.687
4-mer baseline	—	0.588	0.588
Random init	10M	0.596	0.596

Genus Classification (50 held-out genera, linear probe, frozen backbone)

Model	Params	Accuracy	F1
4-mer baseline	—	0.865	0.838
Random init	10M	0.826	0.768
SeqLens v2 Micro	10M	0.798	0.730

Note: Genus classification is composition-dominated (GC content, tetranucleotide frequencies), where k-mer baselines are expected to be competitive. The coding/non-coding task better reflects the model's learned structural and positional representations.

Usage

import torch
from model import SeqLensForMLM
from config import SeqLensConfig, MICRO_CONFIG
from tokenizer import NucleotideTokenizer

# Load model
device = torch.device("cuda")
ckpt = torch.load("seqlens-v2-micro-16k.pt", map_location=device)
model = SeqLensForMLM(MICRO_CONFIG).to(device).to(torch.bfloat16)
model.load_state_dict(ckpt["model"])
model.eval()

# Tokenize a DNA sequence
tokenizer = NucleotideTokenizer(max_len=16384)
seq = "ATGCGATCGATCG..." # your DNA sequence
token_ids = torch.tensor([tokenizer.encode(seq)], dtype=torch.long).to(device)

# Get sequence-level embeddings (for classification tasks)
with torch.no_grad():
    embeddings = model.get_embeddings(token_ids, pool="attention")  # (1, 256)

# Or get per-position predictions (MLM)
with torch.no_grad():
    output = model(token_ids)
    logits = output["logits"]  # (1, L, 9)

Model Family

Variant	Params	Layers	Dim	Context	Status
Micro	10M	8	256	16K	✅ Released
Base	~100M	12	512	32K	In development
Large	~400M	24	768	64K	Planned

Limitations

• Pre-trained on prokaryotic genomes only — may underperform on eukaryotic tasks
• 16K context may truncate long contigs; longer variants planned
• Current model trained for ~1.2B tokens; extended training may improve performance
• Not validated on community-standard benchmarks (GenomicBenchmarks, GUE) yet

Citation

@misc{seqlens-v2-2026,
  title={SeqLens v2: Compact Genomic Language Models for Microbial Bioinformatics},
  author={SeqSight Team},
  year={2026},
  url={https://huggingface.co/seqSight/seqlens-v2-micro-16k}
}

License

Apache 2.0