seqSight/seqlens-v2-micro-novel-16k

SeqLens v2 Micro Novel 16K

A compact genomic language model pre-trained with a novel recipe — span masking + complexity-weighted MLM loss — on prokaryotic genomes. Achieves 96.2% on coding/non-coding classification with only 10M parameters.

What's Different (Novel Pre-training Recipe)

This model uses the same BiMamba2 architecture as seqlens-v2-micro-16k but a different pre-training recipe:

Component	Standard	Novel (this model)
Masking strategy	Random 15% individual tokens	Span masking (contiguous regions, geometric distribution, mean length 10)
Loss function	Uniform cross-entropy	Complexity-weighted cross-entropy (Shannon entropy weighting, low-complexity regions downweighted to 0.1×)
Architecture	BiMamba2 + sliding-window attn	Same
Data	OpenGenome2 GTDB v220	Same

Why this matters: Random masking teaches the model to predict individual nucleotides from local context — a shallow task that saturates quickly. Span masking forces prediction over contiguous regions (codons, promoters, gene boundaries), learning higher-order structural patterns. Complexity weighting prevents the model from wasting capacity on repetitive DNA (~30% of prokaryotic genomes), focusing learning on biologically informative regions.

Results

Coding vs Non-coding (linear probe, frozen backbone)

Model	Params	Accuracy	F1
SeqLens v2 Micro Novel	10M	0.9623	0.9623
SeqLens v2 Micro Standard	10M	0.9111	0.9111
SeqLens v1	89M	0.6872	0.6871
4-mer baseline	—	0.5879	0.5879
Random init	10M	0.5904	0.5903

Genus Classification (50 held-out genera, linear probe, frozen backbone)

Model	Params	Accuracy	F1
4-mer baseline	—	0.8652	0.8381
SeqLens v2 Micro Novel	10M	0.8371	0.8005
SeqLens v2 Micro Standard	10M	0.7978	0.7300
Random init	10M	0.8371	0.7823

Embedding Clustering

Model	Silhouette	ARI
SeqLens v2 Micro Novel	0.0427	0.4026
SeqLens v2 Micro Standard	0.0298	0.3909

Architecture

Property	Value
Parameters	10.3M
Hidden dimension	256
Layers	8 (BiMamba2) + 2 sliding-window attention (layers 3, 7)
Context length	16,384 tokens (single nucleotide)
Vocabulary	A, T, G, C, N, [CLS], [SEP], [PAD], [MASK] (9 tokens)
SSM	Mamba2 (d_state=64, expand=2, headdim=64)
Attention	Chunked sliding-window, w=512, RoPE
FFN	SwiGLU, 4× expansion
Pooling	Attention-weighted (for sequence-level embeddings)

Training

Pre-training Recipe

Parameter	Value
Objective	Masked Language Modeling
Masking	Span masking (geometric, mean=10, max=200)
Loss	Complexity-weighted cross-entropy (window=64, min_weight=0.1)
Mask rate	15% of nucleotides
Mask composition	80% [MASK], 10% random, 10% keep

Optimizer & Schedule

Parameter	Value
Optimizer	AdamW (β₁=0.9, β₂=0.98, ε=1e-8)
Learning rate	5e-4 (cosine decay to 1e-5)
Warmup	1,000 steps
Weight decay	0.1
Gradient clipping	1.0
Precision	BF16 mixed

Data & Compute

Property	Value
Training data	OpenGenome2 GTDB v220 (113,379 prokaryotic genomes)
Pre-tokenized	15.3M chunks × 16,384 tokens = 248B tokens available
Total steps	20,000
Effective batch	64 (8 per GPU × 8 GPUs)
Hardware	8× NVIDIA A100-SXM4-80GB
Framework	PyTorch 2.6.0, mamba-ssm 2.2.4

Usage

import torch
from model import SeqLensForMLM
from config import MICRO_CONFIG
from tokenizer import NucleotideTokenizer

device = torch.device("cuda")
ckpt = torch.load("seqlens-v2-micro-novel-16k.pt", map_location=device)
model = SeqLensForMLM(MICRO_CONFIG).to(device).to(torch.bfloat16)
model.load_state_dict(ckpt["model"])
model.eval()

tokenizer = NucleotideTokenizer(max_len=16384)
seq = "ATGCGATCG..."
token_ids = torch.tensor([tokenizer.encode(seq)], dtype=torch.long).to(device)

# Sequence-level embedding (for classification, retrieval)
with torch.no_grad():
    emb = model.get_embeddings(token_ids, pool="attention")  # (1, 256)

# Per-position logits (for MLM, motif detection)
with torch.no_grad():
    output = model(token_ids)
    logits = output["logits"]  # (1, L, 9)

Model Family

Variant	Params	Recipe	CDS Acc	Genus Acc	Status
Micro Standard	10M	Random MLM	0.911	0.798	Released
Micro Novel	10M	Span + CW	0.962	0.837	Released
Base	~100M	TBD	—	—	Planned

Citation

@misc{seqlens-v2-novel-2026,
  title={SeqLens v2: Span-Masked Complexity-Weighted Pre-training for Compact Genomic Language Models},
  author={SeqSight Team},
  year={2026},
  url={https://huggingface.co/seqSight/seqlens-v2-micro-novel-16k}
}

License

Apache 2.0