petadex/plasticesm-dualseed

PlasticESM (dual seed)

LoRA adapters that continue the masked-language-model pretraining of biohub/ESMC-300M on petadex/catalytic-orfs-90pid. That corpus holds plastic-degrading enzyme homologs mined from the Logan assembly of the SRA. Built for iGEM Toronto 2026 (project PetaBite). Only the adapters were trained. The ESM-C backbone is frozen and is not included here.

Read the downstream results before you use it. The adaptation worked as an MLM, but it did not help any activity-prediction task we tested.

What is in this repo

file	contents
lora_adapters.safetensors	240 tensors, 14,745,600 parameters (59 MB)
lora_manifest.json	format petabite-lora-v1: rank, alpha, targets, and every tensor's shape
config.yaml	the resolved training config as it ran (the paths are cluster paths)
requirements.txt	the exact environment (Compute Canada wheels, so the +computecanada suffixes will not install elsewhere)
tokenizer*.json, special_tokens_map.json	the unchanged ESM-C tokenizer

These adapters are not in PEFT format. ESM-C fuses its LayerNorm into the QKV and MLP projections, so PEFT cannot wrap them. The adapters are loaded with the petabite package instead.

Loading

git clone https://github.com/petadex/petabite && cd petabite
pip install -e .          # needs esm>=3.4.1

import json
from huggingface_hub import snapshot_download
from transformers import AutoModelForMaskedLM, AutoTokenizer
from petabite.model_module.lora import LoraConfig, apply_lora, load_lora

path = snapshot_download("petadex/plasticesm-dualseed")
cfg = json.load(open(f"{path}/lora_manifest.json"))["config"]

model = AutoModelForMaskedLM.from_pretrained("biohub/ESMC-300M", trust_remote_code=True)
apply_lora(model, LoraConfig(r=cfg["r"], alpha=cfg["alpha"], dropout=cfg["dropout"],
                             targets=tuple(cfg["targets"])))
load_lora(model, path, strict=True)   # raises if keys or shapes do not match
model.eval()

tokenizer = AutoTokenizer.from_pretrained("biohub/ESMC-300M")

scripts/verify_checkpoint.py <dir> in the code repo checks three things: that the checkpoint is complete, that it reloads exactly, and that every adapter has moved off its zero initialisation.

Training

base	biohub/ESMC-300M, frozen
LoRA	r 32, alpha 64, dropout 0.01, on qkv, out_proj, fc1, fc2 in all 30 blocks
data	two independent draws (seeds 0 and 1) of one member per 60%-identity cluster of the 90%-identity centroids
sequences	3.63M seen, 2,494,836 distinct (48.7% of the 60% clusters are singletons, so both draws share those)
schedule	1 pass, 18,873 steps, effective batch 192, max length 512, mask rate 0.15, lr 1e-4 cosine, 5% warmup, bf16
compute	10 h 50 m on one GPU, SLURM job 6334470
code	petadex/petabite@979afbc

Result on held-out MLM loss

run	steps	eval loss	perplexity
single draw (twice, independent jobs)	9,437	1.69644 / 1.69640	5.454
two draws (this model)	18,873	1.68426	5.388

The two single-draw runs agree to 4e-5 nats. The 0.0122-nat gain from a second draw is therefore real, about 279 times that run-to-run spread. It is also small: 1.2% of perplexity for 2.05 times the compute. The checkpoint passed all three checks. None of the 240 adapters is still at its initialisation.

Downstream results

We evaluated this model against stock ESM-C 300M on 7 enzyme-activity targets (e.g. kcat, km, kcat/Km, melting temperature and PET monomer release). The tests used grouped cross-validation over homology components and ran on 1,137 labelled sequences.

• It does not beat stock ESM-C on any target, and on several it is worse.
• Neither backbone beats sequence length or amino-acid composition, at any layer, under any of the seven modelling approaches we tried.
• The likely reason: only 53 of the 1,137 labelled sequences (4.7%) appear in the pretraining corpus. The labels are mostly engineered variants of a few scaffolds, while the corpus is natural metagenomic diversity.

The full analysis, including every negative result and the evaluation fixes, is in the Stage 1 notebook.

Known issue: the corpus overlaps the evaluation set

This model was trained on the corpus as built before petadex/petabite#8. That version excluded labelled sequences only when they were themselves 90%-identity centroids, which caught about 120 of 1,219. The 60%-identity clusters of the rest stayed in, so the model may have seen sequences closely related to the evaluation set. Overlap like this could only flatter the model on evaluation, and the model still did not beat the controls. Even so, do not read its scores on those labels as out-of-distribution results. A retrain on the corrected corpus would fix this.

Previous version

petadex/esmc-300m-catalytic-lora held an earlier adapter that was saved incompletely (a fused-LoRA save bug kept 12.5% of the trained tensors). It is retired; use this repo instead.

Citation

If you use this model, cite ESM-C as given on the biohub/ESMC-300M model card.