llama-3-2-1b-sft

This model is a fine-tuned version of NousResearch/Llama-3.2-1B on the HuggingFaceH4/ultrachat_200k dataset. It achieves the following results on the evaluation set:

• Loss: 1.2759

See the training yaml https://github.com/wassname/SimPO/blob/main/training_configs/llama-3-2-1b-base-sft.yaml

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• distributed_type: multi-GPU
• gradient_accumulation_steps: 4
• total_train_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss
1.3663	0.0534	200	1.3955
1.3413	0.1069	400	1.3722
1.365	0.1603	600	1.3632
1.33	0.2138	800	1.3532
1.3219	0.2672	1000	1.3463
1.3355	0.3207	1200	1.3391
1.334	0.3741	1400	1.3305
1.3183	0.4276	1600	1.3233
1.334	0.4810	1800	1.3161
1.3013	0.5345	2000	1.3087
1.3156	0.5879	2200	1.3016
1.3092	0.6414	2400	1.2953
1.2518	0.6948	2600	1.2895
1.2617	0.7483	2800	1.2846
1.3041	0.8017	3000	1.2809
1.3102	0.8552	3200	1.2781
1.2675	0.9086	3400	1.2765
1.2978	0.9621	3600	1.2759

Framework versions

• Transformers 4.45.1
• Pytorch 2.4.1+cu121
• Datasets 3.0.1
• Tokenizers 0.20.0