final-ft__bert-base-spanish-wwm-uncased__70k-ultrasounds

This model is a fine-tuned version of dccuchile/bert-base-spanish-wwm-uncased on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.5422

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 64
• eval_batch_size: 64
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 256
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 20
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
No log	0.9990	254	0.9709
No log	1.9980	508	0.8043
No log	2.9971	762	0.7331
1.013	4.0	1017	0.6860
1.013	4.9990	1271	0.6597
1.013	5.9980	1525	0.6345
1.013	6.9971	1779	0.6280
0.6672	8.0	2034	0.5954
0.6672	8.9990	2288	0.5987
0.6672	9.9980	2542	0.5802
0.6672	10.9971	2796	0.5792
0.6055	12.0	3051	0.5632
0.6055	12.9990	3305	0.5713
0.6055	13.9980	3559	0.5545
0.6055	14.9971	3813	0.5434
0.5735	16.0	4068	0.5475
0.5735	16.9990	4322	0.5471
0.5735	17.9980	4576	0.5377
0.5735	18.9971	4830	0.5403
0.5574	19.9803	5080	0.5422

Framework versions

• Transformers 4.40.1
• Pytorch 2.2.1+cu121
• Datasets 2.19.0
• Tokenizers 0.19.1