pretoxtm-ner
This model is a fine-tuned version of dmis-lab/biobert-v1.1 on javicorvi/pretoxtm-dataset. It achieves the following results on the evaluation set:
- Loss: 0.2722
- Study Test: {'precision': 0.8222222222222222, 'recall': 0.8763157894736842, 'f1': 0.8484076433121018, 'number': 380}
- Manifestation: {'precision': 0.841025641025641, 'recall': 0.9265536723163842, 'f1': 0.8817204301075269, 'number': 177}
- Finding: {'precision': 0.7870485678704857, 'recall': 0.8154838709677419, 'f1': 0.8010139416983523, 'number': 775}
- Specimen: {'precision': 0.7793427230046949, 'recall': 0.8469387755102041, 'f1': 0.8117359413202934, 'number': 392}
- Dose: {'precision': 0.9595959595959596, 'recall': 0.9726962457337884, 'f1': 0.9661016949152542, 'number': 293}
- Dose Qualification: {'precision': 0.8787878787878788, 'recall': 0.8787878787878788, 'f1': 0.8787878787878788, 'number': 33}
- Sex: {'precision': 0.9279279279279279, 'recall': 0.9809523809523809, 'f1': 0.9537037037037037, 'number': 105}
- Group: {'precision': 0.8913043478260869, 'recall': 0.8817204301075269, 'f1': 0.8864864864864864, 'number': 93}
- Precision: 0.8298
- Recall: 0.8719
- F1: 0.8503
- Accuracy: 0.9530
Model description
PretoTM NER is a model developed to recognize relevant entities associated with treatment-related findings in preclinical toxicology.
Training and evaluation data
The model was trained on javicorvi/pretoxtm-dataset.
The dataset is divided in train, validation and test.
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 5.760003080365119e-05
- train_batch_size: 4
- eval_batch_size: 8
- seed: 4
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 7
Training results
| Training Loss | Epoch | Step | Validation Loss | Study Test | Manifestation | Finding | Specimen | Dose | Dose Qualification | Sex | Group | Precision | Recall | F1 | Accuracy |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0.3365 | 1.0 | 514 | 0.2023 | {'precision': 0.740909090909091, 'recall': 0.8578947368421053, 'f1': 0.7951219512195122, 'number': 380} | {'precision': 0.7874396135265701, 'recall': 0.9209039548022598, 'f1': 0.8489583333333334, 'number': 177} | {'precision': 0.7055214723926381, 'recall': 0.7419354838709677, 'f1': 0.7232704402515724, 'number': 775} | {'precision': 0.7312072892938497, 'recall': 0.8188775510204082, 'f1': 0.772563176895307, 'number': 392} | {'precision': 0.9243986254295533, 'recall': 0.9180887372013652, 'f1': 0.9212328767123289, 'number': 293} | {'precision': 0.8, 'recall': 0.8484848484848485, 'f1': 0.823529411764706, 'number': 33} | {'precision': 0.9074074074074074, 'recall': 0.9333333333333333, 'f1': 0.9201877934272301, 'number': 105} | {'precision': 0.7441860465116279, 'recall': 0.6881720430107527, 'f1': 0.7150837988826816, 'number': 93} | 0.7617 | 0.8203 | 0.7899 | 0.9397 |
| 0.1325 | 2.0 | 1028 | 0.2012 | {'precision': 0.7920792079207921, 'recall': 0.8421052631578947, 'f1': 0.8163265306122449, 'number': 380} | {'precision': 0.8488372093023255, 'recall': 0.8248587570621468, 'f1': 0.836676217765043, 'number': 177} | {'precision': 0.7163289630512515, 'recall': 0.775483870967742, 'f1': 0.7447335811648079, 'number': 775} | {'precision': 0.723175965665236, 'recall': 0.8596938775510204, 'f1': 0.7855477855477856, 'number': 392} | {'precision': 0.9013157894736842, 'recall': 0.9351535836177475, 'f1': 0.9179229480737018, 'number': 293} | {'precision': 0.8181818181818182, 'recall': 0.8181818181818182, 'f1': 0.8181818181818182, 'number': 33} | {'precision': 0.911504424778761, 'recall': 0.9809523809523809, 'f1': 0.944954128440367, 'number': 105} | {'precision': 0.8645833333333334, 'recall': 0.8924731182795699, 'f1': 0.8783068783068784, 'number': 93} | 0.7792 | 0.8412 | 0.8090 | 0.9450 |
| 0.0733 | 3.0 | 1542 | 0.2101 | {'precision': 0.7451403887688985, 'recall': 0.9078947368421053, 'f1': 0.8185053380782918, 'number': 380} | {'precision': 0.7922705314009661, 'recall': 0.9265536723163842, 'f1': 0.8541666666666666, 'number': 177} | {'precision': 0.7481840193704601, 'recall': 0.7974193548387096, 'f1': 0.7720174890693317, 'number': 775} | {'precision': 0.7676056338028169, 'recall': 0.8341836734693877, 'f1': 0.7995110024449877, 'number': 392} | {'precision': 0.9276315789473685, 'recall': 0.962457337883959, 'f1': 0.9447236180904522, 'number': 293} | {'precision': 0.7941176470588235, 'recall': 0.8181818181818182, 'f1': 0.8059701492537314, 'number': 33} | {'precision': 0.9292035398230089, 'recall': 1.0, 'f1': 0.9633027522935781, 'number': 105} | {'precision': 0.8695652173913043, 'recall': 0.8602150537634409, 'f1': 0.8648648648648649, 'number': 93} | 0.7903 | 0.8665 | 0.8266 | 0.9484 |
| 0.0431 | 4.0 | 2056 | 0.2260 | {'precision': 0.8477157360406091, 'recall': 0.8789473684210526, 'f1': 0.8630490956072351, 'number': 380} | {'precision': 0.8190954773869347, 'recall': 0.9209039548022598, 'f1': 0.8670212765957446, 'number': 177} | {'precision': 0.7653562653562653, 'recall': 0.8038709677419354, 'f1': 0.7841409691629955, 'number': 775} | {'precision': 0.7897196261682243, 'recall': 0.8622448979591837, 'f1': 0.824390243902439, 'number': 392} | {'precision': 0.9459459459459459, 'recall': 0.9556313993174061, 'f1': 0.9507640067911713, 'number': 293} | {'precision': 0.8055555555555556, 'recall': 0.8787878787878788, 'f1': 0.8405797101449276, 'number': 33} | {'precision': 0.9203539823008849, 'recall': 0.9904761904761905, 'f1': 0.9541284403669724, 'number': 105} | {'precision': 0.8842105263157894, 'recall': 0.9032258064516129, 'f1': 0.8936170212765957, 'number': 93} | 0.8232 | 0.8697 | 0.8458 | 0.9515 |
| 0.0282 | 5.0 | 2570 | 0.2492 | {'precision': 0.835820895522388, 'recall': 0.8842105263157894, 'f1': 0.8593350383631714, 'number': 380} | {'precision': 0.8333333333333334, 'recall': 0.9322033898305084, 'f1': 0.8800000000000001, 'number': 177} | {'precision': 0.7820197044334976, 'recall': 0.8193548387096774, 'f1': 0.800252047889099, 'number': 775} | {'precision': 0.785377358490566, 'recall': 0.8494897959183674, 'f1': 0.8161764705882352, 'number': 392} | {'precision': 0.9627118644067797, 'recall': 0.9692832764505119, 'f1': 0.9659863945578231, 'number': 293} | {'precision': 0.8235294117647058, 'recall': 0.8484848484848485, 'f1': 0.8358208955223881, 'number': 33} | {'precision': 0.9285714285714286, 'recall': 0.9904761904761905, 'f1': 0.9585253456221199, 'number': 105} | {'precision': 0.9120879120879121, 'recall': 0.8924731182795699, 'f1': 0.9021739130434783, 'number': 93} | 0.8311 | 0.8754 | 0.8527 | 0.9528 |
| 0.0125 | 6.0 | 3084 | 0.2668 | {'precision': 0.830423940149626, 'recall': 0.8763157894736842, 'f1': 0.852752880921895, 'number': 380} | {'precision': 0.839572192513369, 'recall': 0.8870056497175142, 'f1': 0.8626373626373628, 'number': 177} | {'precision': 0.7724477244772447, 'recall': 0.8103225806451613, 'f1': 0.7909319899244331, 'number': 775} | {'precision': 0.7617977528089888, 'recall': 0.8647959183673469, 'f1': 0.8100358422939069, 'number': 392} | {'precision': 0.9726962457337884, 'recall': 0.9726962457337884, 'f1': 0.9726962457337884, 'number': 293} | {'precision': 0.875, 'recall': 0.8484848484848485, 'f1': 0.8615384615384615, 'number': 33} | {'precision': 0.9279279279279279, 'recall': 0.9809523809523809, 'f1': 0.9537037037037037, 'number': 105} | {'precision': 0.8913043478260869, 'recall': 0.8817204301075269, 'f1': 0.8864864864864864, 'number': 93} | 0.8235 | 0.8697 | 0.8460 | 0.9529 |
| 0.006 | 7.0 | 3598 | 0.2722 | {'precision': 0.8222222222222222, 'recall': 0.8763157894736842, 'f1': 0.8484076433121018, 'number': 380} | {'precision': 0.841025641025641, 'recall': 0.9265536723163842, 'f1': 0.8817204301075269, 'number': 177} | {'precision': 0.7870485678704857, 'recall': 0.8154838709677419, 'f1': 0.8010139416983523, 'number': 775} | {'precision': 0.7793427230046949, 'recall': 0.8469387755102041, 'f1': 0.8117359413202934, 'number': 392} | {'precision': 0.9595959595959596, 'recall': 0.9726962457337884, 'f1': 0.9661016949152542, 'number': 293} | {'precision': 0.8787878787878788, 'recall': 0.8787878787878788, 'f1': 0.8787878787878788, 'number': 33} | {'precision': 0.9279279279279279, 'recall': 0.9809523809523809, 'f1': 0.9537037037037037, 'number': 105} | {'precision': 0.8913043478260869, 'recall': 0.8817204301075269, 'f1': 0.8864864864864864, 'number': 93} | 0.8298 | 0.8719 | 0.8503 | 0.9530 |
Framework versions
- Transformers 4.39.3
- Pytorch 2.2.1+cu121
- Datasets 2.18.0
- Tokenizers 0.15.2
- Downloads last month
- 40
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social
visibility and check back later, or deploy to Inference Endpoints (dedicated)
instead.
Model tree for javicorvi/pretoxtm-ner
Base model
dmis-lab/biobert-v1.1