multimolecule/ablang-heavy

AbLang

Pre-trained antibody language model using a masked language modeling (MLM) objective.

Disclaimer

This is an UNOFFICIAL implementation of AbLang: an antibody language model for completing antibody sequences by Tobias H. Olsen, et al.

The OFFICIAL repository of AbLang is at oxpig/AbLang.

The MultiMolecule team has confirmed that the provided model and checkpoints are producing the same intermediate representations as the original implementation.

The team releasing AbLang did not write this model card for this model so this model card has been written by the MultiMolecule team.

Model Details

AbLang v1 is an encoder-only Transformer trained on antibody sequences from the Observed Antibody Space (OAS). The official release provides separate heavy-chain and light-chain checkpoints. Both variants use the same architecture and vocabulary, but they were trained on chain-specific data and are represented as separate MultiMolecule variants.

Variants

• multimolecule/ablang-heavy: AbLang v1 trained on heavy-chain antibody sequences.
• multimolecule/ablang-light: AbLang v1 trained on light-chain antibody sequences.

Model Specification

Variant	Chain Type	Num Layers	Hidden Size	Num Heads	Intermediate Size	Num Parameters (M)	FLOPs (G)	MACs (G)	Max Num Tokens
AbLang-Heavy	Heavy	12	768	12	3072	85.83	28.18	14.06	159
AbLang-Light	Light

Links

• Code: multimolecule.ablang
• Data: Observed Antibody Space
• Paper: AbLang: an antibody language model for completing antibody sequences
• Developed by: Tobias H. Olsen, Iain H. Moal, Charlotte M. Deane
• Model type: Encoder-only Transformer for antibody masked language modeling
• Original Repository: oxpig/AbLang

Usage

The model file depends on the multimolecule library. You can install it using pip:

pip install multimolecule

Direct Use

Masked Language Modeling

You can use this model directly with a pipeline for masked language modeling:

import multimolecule  # you must import multimolecule to register models
from transformers import pipeline

predictor = pipeline("fill-mask", model="multimolecule/ablang-heavy")
output = predictor("EVQLVESGGGLVQPGGSLRLSCAASGFTFSSY<mask>MSWVRQAPGKGLEWVSA")

Downstream Use

Extract Features

Here is how to use this model to get the features of a given antibody sequence in PyTorch:

from multimolecule import AbLangModel, ProteinTokenizer


tokenizer = ProteinTokenizer.from_pretrained("multimolecule/ablang-heavy")
model = AbLangModel.from_pretrained("multimolecule/ablang-heavy")

text = "EVQLVESGGGLVQPGGSLRLSCAASGFTFSSYAMSWVRQAPGKGLEWVSA"
input = tokenizer(text, return_tensors="pt")

output = model(**input)

Sequence Classification / Regression

This model is not fine-tuned for any specific task. You will need to fine-tune the model on a downstream task to use it for sequence classification or regression.

Here is how to use this model as backbone to fine-tune for a sequence-level task in PyTorch:

import torch
from multimolecule import AbLangForSequencePrediction, ProteinTokenizer


tokenizer = ProteinTokenizer.from_pretrained("multimolecule/ablang-heavy")
model = AbLangForSequencePrediction.from_pretrained("multimolecule/ablang-heavy")

text = "EVQLVESGGGLVQPGGSLRLSCAASGFTFSSYAMSWVRQAPGKGLEWVSA"
input = tokenizer(text, return_tensors="pt")
label = torch.tensor([1])

output = model(**input, labels=label)

Token Classification / Regression

This model is not fine-tuned for any specific task. You will need to fine-tune the model on a downstream task to use it for token classification or regression.

Here is how to use this model as backbone to fine-tune for a residue-level task in PyTorch:

import torch
from multimolecule import AbLangForTokenPrediction, ProteinTokenizer


tokenizer = ProteinTokenizer.from_pretrained("multimolecule/ablang-heavy")
model = AbLangForTokenPrediction.from_pretrained("multimolecule/ablang-heavy")

text = "EVQLVESGGGLVQPGGSLRLSCAASGFTFSSYAMSWVRQAPGKGLEWVSA"
input = tokenizer(text, return_tensors="pt")
label = torch.randint(2, (len(text), ))

output = model(**input, labels=label)

Training Details

AbLang was trained with masked language modeling (MLM) as the pre-training objective.

Training Data

AbLang was trained on antibody sequences from the Observed Antibody Space. The heavy-chain model was trained on 14,126,724 sequences, and the light-chain model was trained on 187,068 sequences.

Training Procedure

Pre-training

The heavy-chain and light-chain checkpoints were trained separately on chain-specific OAS sequences. Please refer to the original paper for details on the training setup.

Citation

@article{olsen2022ablang,
  title   = {AbLang: an antibody language model for completing antibody sequences},
  author  = {Olsen, Tobias H. and Moal, Iain H. and Deane, Charlotte M.},
  journal = {Bioinformatics Advances},
  volume  = {2},
  number  = {1},
  pages   = {vbac046},
  year    = {2022},
  doi     = {10.1093/bioadv/vbac046},
  url     = {https://doi.org/10.1093/bioadv/vbac046},
}

The artifacts distributed in this repository are part of the MultiMolecule project. If MultiMolecule supports your research, please cite the MultiMolecule project as follows:

@software{chen_2024_12638419,
  author    = {Chen, Zhiyuan and Zhu, Sophia Y.},
  title     = {MultiMolecule},
  doi       = {10.5281/zenodo.12638419},
  publisher = {Zenodo},
  url       = {https://doi.org/10.5281/zenodo.12638419},
  year      = 2024,
  month     = may,
  day       = 4
}

Contact

Please use GitHub issues of MultiMolecule for any questions or comments on the model card.

Please contact the authors of the AbLang paper for questions or comments on the paper/model.

License

This model implementation is licensed under the GNU Affero General Public License.

For additional terms and clarifications, please refer to our License FAQ.

SPDX-License-Identifier: AGPL-3.0-or-later