AntiBERTy

AntiBERTy is an antibody-specific transformer language model pre-trained on 558M natural antibody sequences from the Observed Antibody Space, as described in Deciphering antibody affinity maturation with language models and weakly supervised learning. It is an 8-layer BERT encoder (512 hidden units, 8 attention heads, 26M parameters) trained with masked language modelling together with species and chain-type classification heads, and provides the sequence features used by IgFold for antibody structure prediction.

Files

File	Description
model.safetensors	Model weights (fp32, 104 MB)
config.json	transformers BertConfig for the encoder
vocab.txt	25-token vocabulary: [PAD] [UNK] [CLS] [SEP] [MASK] + 20 amino acids, one token per residue

Usage

Install the antiberty package; the weights are downloaded from this repository on first use and cached.

pip install antiberty
antiberty classify EVQLVQSGPEVKKPGTSVKVSCKASGFTFMSSAVQWVRQARGQRLEWIGWIVIGSGNTNYAQKFQERVTITRDMSTSTAYMELSSLRSEDTAVYYCAAPYCSSISCNDGFDIWGQGTMVTVS

from antiberty import AntiBERTyRunner

antiberty = AntiBERTyRunner()
embeddings = antiberty.embed(["EVQLVQSGPEVKKPGTSVKVSCKASGFTFMSSAVQWVRQARGQRLEWIGWIVIGSGNTNYAQKFQERVTITRDMSTSTAYMELSSLRSEDTAVYYCAAPYCSSISCNDGFDIWGQGTMVTVS"])
species, chains = antiberty.classify([...])
filled = antiberty.fill_masks(["QVQLQESGGGLVQAGGSLTLSCAVSG__FSNYAMG"])

The model class (antiberty.AntiBERTy, a transformers.BertPreTrainedModel with the extra classification heads) and the tokenizer live in the package; the weights can also be loaded with AntiBERTy.from_pretrained(<local snapshot>, attn_implementation="eager").

Intended use and limitations

AntiBERTy expects antibody variable-domain sequences (roughly 100-130 residues per chain, maximum 510). It was trained on heavy and light chains from human, mouse, rat, rabbit, rhesus and camelid repertoires; outputs for other proteins or full-length antibody chains are not meaningful.

Citation

@article{ruffolo2021deciphering,
    title = {Deciphering antibody affinity maturation with language models and weakly supervised learning},
    author = {Ruffolo, Jeffrey A and Gray, Jeffrey J and Sulam, Jeremias},
    journal = {arXiv preprint arXiv:2112.07782},
    year= {2021}
}