Llama-3.2-3B-Danoia

Der er ikke meget at sige andet end at den kan dansk.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 2
• eval_batch_size: 1
• seed: 222
• gradient_accumulation_steps: 8
• total_train_batch_size: 16
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.01
• num_epochs: 3

Training results

Training Loss	Epoch	Step	Validation Loss
1.0895	0.2103	500	1.0488
1.0893	0.4205	1000	0.9952
0.864	0.6308	1500	0.9645
0.9665	0.8411	2000	0.9406
0.9387	1.0514	2500	0.9242
0.7996	1.2617	3000	0.9126
0.7904	1.4720	3500	0.9005
0.9745	1.6822	4000	0.8926
1.0152	1.8925	4500	0.8859
0.7676	2.1028	5000	0.8821
0.8127	2.3131	5500	0.8791
0.9498	2.5234	6000	0.8770
0.795	2.7336	6500	0.8758
0.8029	2.9439	7000	0.8758

Framework versions

• PEFT 0.11.1
• Transformers 4.46.1
• Pytorch 2.5.1
• Datasets 2.20.0
• Tokenizers 0.20.3