BERTModified-rawbert-finetuned-wikitext-test

This model is a fine-tuned version of distilbert-base-uncased on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 21.1324
• Precision: 0.0411
• Recall: 0.0411
• F1: 0.0411
• Accuracy: 0.0411

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 20

Training results

Training Loss	Epoch	Step	Validation Loss	Precision	Recall	F1	Accuracy
21.2329	1.0	25	21.1260	0.0157	0.0157	0.0157	0.0157
18.2402	2.0	50	20.8705	0.0181	0.0181	0.0181	0.0181
15.7819	3.0	75	20.8017	0.0206	0.0206	0.0206	0.0206
13.5492	4.0	100	20.7470	0.0206	0.0206	0.0206	0.0206
11.836	5.0	125	20.7319	0.0254	0.0254	0.0254	0.0254
10.306	6.0	150	20.7540	0.0314	0.0314	0.0314	0.0314
9.0142	7.0	175	20.7665	0.0363	0.0363	0.0363	0.0363
7.991	8.0	200	20.8323	0.0363	0.0363	0.0363	0.0363
7.0936	9.0	225	20.9107	0.0387	0.0387	0.0387	0.0387
6.3742	10.0	250	20.9569	0.0399	0.0399	0.0399	0.0399
5.7236	11.0	275	20.9811	0.0375	0.0375	0.0375	0.0375
5.3262	12.0	300	21.0331	0.0435	0.0435	0.0435	0.0435
4.8222	13.0	325	21.0361	0.0387	0.0387	0.0387	0.0387
4.5049	14.0	350	21.0610	0.0363	0.0363	0.0363	0.0363
4.1877	15.0	375	21.0827	0.0387	0.0387	0.0387	0.0387
3.9705	16.0	400	21.1020	0.0435	0.0435	0.0435	0.0435
3.8091	17.0	425	21.0863	0.0435	0.0435	0.0435	0.0435
3.5978	18.0	450	21.1236	0.0447	0.0447	0.0447	0.0447
3.5991	19.0	475	21.1211	0.0435	0.0435	0.0435	0.0435
3.5433	20.0	500	21.1236	0.0435	0.0435	0.0435	0.0435

Framework versions

• Transformers 4.39.1
• Pytorch 2.2.1+cu121
• Datasets 2.18.0
• Tokenizers 0.15.2