wkdghdus23/astra-bert-mlm

ASTRA: Pre-trained BERT (MLM)

This model is part of the ASTRA (Advanced Solvation Transformer for Rational Additives) framework. It is a BERT-based Masked Language Model (MLM) heavily pre-trained on a large-scale dataset of canonical SMILES strings. By predicting masked tokens within molecular sequences, this model has learned the fundamental chemical syntax, atomic connectivity, and structural representations required for advanced cheminformatics tasks.

This pre-trained encoder serves as the foundational base model for subsequent downstream property prediction tasks (e.g., LUMO and Binding Energy regression) within the ASTRA framework.

Model Details

• Architecture: BERT (MaskedLM)
• Stage: Pre-trained (Base Model)
• Task: Masked Language Modeling (MLM)
• Data: Large-scale unlabelled canonical SMILES (~4M)

Usage

from astra.tokenizer import initial_bert_tokenizer_with_vocabulary
from astra.model import BertForMLM

# Load the tokenizer and model
vocab_file = "./vocab.txt"
tokenizer = initial_bert_tokenizer_with_vocabulary(path=vocab_file)

pretrained_path = "<path/to/pretrained/model>" # Replace with your actual paths
model = BertForMLM.from_pretrained_model(pretrained_path, tokenizer)

More Information

For more details on data preparation, downstream fine-tuning, and the full active learning loop, please visit our GitHub Repository.