CompoundT5 / README.md
sagawa's picture
Update README.md
129b06b
metadata
license: mit
datasets:
  - sagawa/ZINC-canonicalized
metrics:
  - accuracy
model-index:
  - name: ZINC-deberta
    results:
      - task:
          name: Masked Language Modeling
          type: fill-mask
        dataset:
          name: sagawa/ZINC-canonicalized
          type: sagawa/ZINC-canonicalized
        metrics:
          - name: Accuracy
            type: accuracy
            value: 0.9497212171554565

CompoundT5

This model is a re-pretrained version of google/t5-v1_1-base on the sagawa/ZINC-canonicalized dataset. It achieves the following results on the evaluation set:

  • Loss: 0.1202
  • Accuracy: 0.9497

Model description

We trained t5 on SMILES from ZINC using masked-language modeling (MLM). Its tokenizer is also trained on ZINC.

Intended uses & limitations

This model can be used to predict molecules' properties, reactions, or interactions with proteins by changing the way of finetuning. As an example, We finetuned this model to predict products. The model is here, and you can use the demo here. Using its encoder, we trained a regression model to predict a reaction yield. You can use this demo here.

Training and evaluation data

We downloaded ZINC data and canonicalized them using RDKit. Then, we dropped duplicates. The total number of data is 22992522, and they were randomly split into train:validation=10:1.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 5e-03
  • train_batch_size: 30
  • eval_batch_size: 32
  • seed: 42
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • num_epochs: 30.0

Training results

Training Loss Step Accuracy Validation Loss
0.2471 25000 0.9843 0.2226
0.1871 50000 0.9314 0.1783
0.1791 75000 0.9371 0.1619
0.1596 100000 0.9401 0.1520
0.1522 125000 0.9422 0.1449
0.1435 150000 0.9436 0.1404
0.1421 175000 0.9447 0.1368
0.1398 200000 0.9459 0.1322
0.1297 225000 0.9466 0.1299
0.1324 250000 0.9473 0.1268
0.1257 275000 0.9483 0.1244
0.1266 300000 0.9491 0.1216
0.1301 325000 0.9497 0.1204