metadata

license: mit
datasets:
  - sagawa/ZINC-canonicalized
metrics:
  - accuracy
model-index:
  - name: ZINC-deberta
    results:
      - task:
          name: Masked Language Modeling
          type: fill-mask
        dataset:
          name: sagawa/ZINC-canonicalized
          type: sagawa/ZINC-canonicalized
        metrics:
          - name: Accuracy
            type: accuracy
            value: 0.9497212171554565

ZINC-t5

This model is a fine-tuned version of google/t5-v1_1-base on the sagawa/ZINC-canonicalized dataset. It achieves the following results on the evaluation set:

Loss: 0.1202
Accuracy: 0.9497

Model description

We trained t5 on SMILES from ZINC using the task of masked-language modeling (MLM). Its tokenizer is also trained on ZINC.

Intended uses & limitations

This model can be used for the prediction of molecules' properties, reactions, or interactions with proteins by changing the way of finetuning. As an example, We finetuned this model to predict products. Model is here, and you can use the demo here. Using its encoder, we trained a regression model to predict a reaction yield. You can use this demo here.

Training and evaluation data

We downloaded ZINC data and canonicalized them using RDKit. Then, we droped duplicates. The total number of data is 22992522, and they were randomly split into train:validation=10:1.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 5e-03
train_batch_size: 30
eval_batch_size: 32
seed: 42
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: linear
num_epochs: 30.0

Training results

Training Loss	Step	Accuracy	Validation Loss
0.2226	25000	0.9843	0.2226
0.1783	50000	0.9314	0.1783
0.1619	75000	0.9371	0.1619
0.1520	100000	0.9401	0.1520
0.1449	125000	0.9422	0.1449
0.1404	150000	0.9436	0.1404
0.1368	175000	0.9447	0.1368
0.1322	200000	0.9459	0.1322
0.1299	225000	0.9466	0.1299
0.1268	250000	0.9473	0.1268
0.1244	275000	0.9483	0.1244
0.1216	300000	0.9491	0.1216
0.1204	325000	0.9497	0.1204