FungiHelixSeek

A Hybrid Linear Attention-Transformer Mixture-of-Experts (MoE) DNA foundation model for Fungi — pretrained with bidirectional masked language modeling at single-nucleotide resolution on fungal genomes.

Property	Value
Total parameters	263M
Active parameters per token	~112M (top-2 of 8 routed experts + 1 shared)
Hidden size	512
Layers	14 hybrid (6 Transformer + 6 KDA linear-attention + 2 MLA)
Attention heads	16
Max context	8,192 bp
Vocabulary	11 tokens (A, T, C, G, N + specials, 1 nt = 1 token)
Pretraining	Masked language modeling

Benchmark highlight

First place on 11 of 12 fungal epigenomic tasks against 8 leading DNA foundation models (NT-v2, GENERanno, PlantNT-6mer, DNABERT-2, DNAMamba, ProkBERT, GENA-LM-yeast, Evo 2): rank points 107/108, mean performance 0.826 (+2.1 pp over runner-up). See the GitHub repository for the full comparison.

Usage

from transformers import AutoModelForMaskedLM, AutoTokenizer
import torch

model_path = "zhangtaolab/FungiHelixSeek"
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
model = AutoModelForMaskedLM.from_pretrained(model_path, trust_remote_code=True).cuda().eval()

seq = "ATCGGCTAAGCCT<mask>GTCAGTTACCGATCG"
enc = tokenizer(seq, return_tensors="pt"); enc = {k: v.cuda() for k, v in enc.items()}
with torch.no_grad():
    logits = model(**enc).logits
pos = (enc["input_ids"][0] == tokenizer.mask_token_id).nonzero()[0]
print(tokenizer.convert_ids_to_tokens(logits[0, pos].argmax(-1)))  # → ['T']

Task models

Fine-tuned models built on this base: FungiHelixSeek-Taxonomy (6-level ITS classification), FungiHelixSeek-Anno (gene structure prediction), and 12 histone-mark classifiers (FungiHelixSeek-H3 … FungiHelixSeek-H3K27me3) — see the FungiHelixSeek collection.

License

CC-BY-NC 4.0 (research use; contact authors for commercial licensing).