File size: 2,772 Bytes
56a3a08 3934e14 56a3a08 3934e14 129b06b b57fa29 129b06b 3934e14 129b06b 3934e14 129b06b 6aff882 3934e14 c7aecdc 6aff882 3934e14 6aff882 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 |
---
license: mit
datasets:
- sagawa/ZINC-canonicalized
metrics:
- accuracy
model-index:
- name: ZINC-deberta
results:
- task:
name: Masked Language Modeling
type: fill-mask
dataset:
name: sagawa/ZINC-canonicalized
type: sagawa/ZINC-canonicalized
metrics:
- name: Accuracy
type: accuracy
value: 0.9497212171554565
---
# CompoundT5
This model is a re-pretrained version of [google/t5-v1_1-base](https://huggingface.co/microsoft/deberta-base) on the sagawa/ZINC-canonicalized dataset.
It achieves the following results on the evaluation set:
- Loss: 0.1202
- Accuracy: 0.9497
## Model description
We trained t5 on SMILES from ZINC using masked-language modeling (MLM). Its tokenizer is also trained on ZINC.
## Intended uses & limitations
This model can be used to predict molecules' properties, reactions, or interactions with proteins by changing the way of finetuning.
As an example, We finetuned this model to predict products. The model is [here](https://huggingface.co/sagawa/ZINC-t5-productpredicition), and you can use the demo [here](https://huggingface.co/spaces/sagawa/predictproduct-t5).
Using its encoder, we trained a regression model to predict a reaction yield. You can use this demo [here](https://huggingface.co/spaces/sagawa/predictyield-t5).
## Training and evaluation data
We downloaded [ZINC data](https://drive.google.com/drive/folders/1lSPCqh31zxTVEhuiPde7W3rZG8kPgp-z) and canonicalized them using RDKit. Then, we dropped duplicates. The total number of data is 22992522, and they were randomly split into train:validation=10:1.
## Training procedure
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 5e-03
- train_batch_size: 30
- eval_batch_size: 32
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 30.0
### Training results
| Training Loss | Step | Accuracy | Validation Loss |
|:-------------:|:------:|:--------:|:---------------:|
| 0.2471 | 25000 | 0.9843 | 0.2226 |
| 0.1871 | 50000 | 0.9314 | 0.1783 |
| 0.1791 | 75000 | 0.9371 | 0.1619 |
| 0.1596 | 100000 | 0.9401 | 0.1520 |
| 0.1522 | 125000 | 0.9422 | 0.1449 |
| 0.1435 | 150000 | 0.9436 | 0.1404 |
| 0.1421 | 175000 | 0.9447 | 0.1368 |
| 0.1398 | 200000 | 0.9459 | 0.1322 |
| 0.1297 | 225000 | 0.9466 | 0.1299 |
| 0.1324 | 250000 | 0.9473 | 0.1268 |
| 0.1257 | 275000 | 0.9483 | 0.1244 |
| 0.1266 | 300000 | 0.9491 | 0.1216 |
| 0.1301 | 325000 | 0.9497 | 0.1204 | |