petadex/esmc-300m-catalytic-lora

PlasticESM (ESM-C 300M, LoRA, MLM domain-adapted)

PlasticESM is a LoRA adapter that domain-adapts biohub/ESMC-300M to the diversity of plastic-degrading enzymes. It is Stage 1 of the petabite pipeline: continued masked-language-model pretraining on ~18M catalytic ORF homologs mined from the Logan planetary-scale SRA assembly, producing a backbone that "understands" PETase-family sequence space before any activity labels are introduced.

Model Details

• Developed by: petabite / petadex
• Model type: Protein language model (ESM-C encoder) with LoRA adapters
• Objective: Masked language modeling (continued / domain-adaptive pretraining)
• Language(s): Protein sequences (amino-acid tokens)
• Finetuned from: biohub/ESMC-300M
• License: Inherited from the base model biohub/ESMC-300M — confirm before redistribution.

Model Sources

• Base model: https://huggingface.co/biohub/ESMC-300M
• Training dataset: https://huggingface.co/datasets/petadex/catalytic-orfs-90pid

Uses

Direct Use

Domain-adapted backbone for masked-residue infilling and per-residue / per-sequence embeddings of plastic-degrading enzyme candidates.

Downstream Use

• Stage 2 — supervised activity / fitness prediction on labeled PETase variants.
• Stage 3 — conditioned sequence generation of novel candidate enzymes.

Out-of-Scope Use

Not a substitute for experimental validation. Predictions are hypotheses for wet-lab testing, not assurances of catalytic activity, expressibility, or safety.

Bias, Risks, and Limitations

Training sequences are metagenomic ORFs from environmental SRA assemblies, so taxonomic and functional coverage is skewed toward what is abundant in those samples. The 90%-identity centroid dereplication reduces redundancy but does not remove sampling bias. Embeddings and infilled residues reflect the training distribution and may generalize poorly to enzyme families that are rare or absent in the corpus.

How to Get Started with the Model

from peft import PeftModel
from transformers import AutoModelForMaskedLM, AutoTokenizer

base = AutoModelForMaskedLM.from_pretrained("biohub/ESMC-300M", trust_remote_code=True)
model = PeftModel.from_pretrained(base, "petadex/esmc-300m-catalytic-lora").eval()
tokenizer = AutoTokenizer.from_pretrained("biohub/ESMC-300M")

Training Details

Training Data

petadex/catalytic-orfs-90pid: 18,172,960 catalytic ORF sequences (90% sequence-identity centroids, already dereplicated), split 98 / 1 / 1 into train / validation / test. Training uses the Sequence column.

Training Procedure

• Objective: masked language modeling, mlm_probability = 0.15
• Adapters: LoRA r = 32, alpha = 16, dropout = 0.01, bias = "none", applied to attention out_proj (via PEFT) plus the fused QKV / FFN projections (via a custom injection for ESM-C's fused LayerNorm+Linear modules)
• Backbone: frozen; only LoRA matrices are trained
• Max sequence length: 1024
• Epochs: 1
• Batch size: 16 per device, gradient accumulation 1
• Optimizer / schedule: lr 1e-4, cosine schedule, warmup ratio 0.05, weight decay 0.01
• Precision: bf16
• Seed: 42

Evaluation

Held-out validation split (~182k sequences) of the same dataset, evaluated with the masked-LM loss / perplexity. Populate this section with the final numbers from the run.

Framework versions

• PEFT 0.19.1