zlm-fil-ceb_b64_le5_s8000

This model is a fine-tuned version of mikhail-panzo/zlm-fil_b64_le5_s8000 on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.3939

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 16
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 64
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 2000
• training_steps: 8000
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
0.4592	19.8020	500	0.4253
0.4381	39.6040	1000	0.4100
0.4281	59.4059	1500	0.4022
0.4195	79.2079	2000	0.3988
0.4134	99.0099	2500	0.3955
0.4049	118.8119	3000	0.3935
0.4016	138.6139	3500	0.3931
0.3937	158.4158	4000	0.3936
0.3953	178.2178	4500	0.3935
0.3933	198.0198	5000	0.3936
0.3932	217.8218	5500	0.3936
0.3884	237.6238	6000	0.3934
0.3889	257.4257	6500	0.3926
0.3855	277.2277	7000	0.3940
0.3892	297.0297	7500	0.3934
0.3899	316.8317	8000	0.3939

Framework versions

• Transformers 4.41.0.dev0
• Pytorch 2.2.1+cu121
• Datasets 2.19.0
• Tokenizers 0.19.1