sft-fsi-masked-loss

This model was trained from scratch on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.3788

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• distributed_type: multi-GPU
• num_devices: 8
• gradient_accumulation_steps: 8
• total_train_batch_size: 256
• total_eval_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 20

Training results

Training Loss	Epoch	Step	Validation Loss
17.1473	0.5333	1	18.7762
17.1473	1.6	3	9.0847
11.7791	2.6667	5	3.0550
11.7791	3.7333	7	0.6706
11.7791	4.8	9	0.6697
1.4045	5.8667	11	0.5653
1.4045	6.9333	13	0.4982
0.6622	8.0	15	0.4756
0.6622	8.5333	16	0.4777
0.6622	9.6	18	0.4338
0.5586	10.6667	20	0.3788

Framework versions

• Transformers 4.40.1
• Pytorch 2.3.0+cu121
• Datasets 2.19.0
• Tokenizers 0.19.1