thiomajid/hausa-gemma2

hausa_lm

This model is a fine-tuned version of google/gemma-2b on the None dataset. It achieves the following results on the evaluation set:

• Loss: 5.3143

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 10
• eval_batch_size: 10
• seed: 42
• gradient_accumulation_steps: 3
• total_train_batch_size: 30
• optimizer: Use adamw_torch_fused with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 5
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
No log	0.3	100	8.1906
24.3915	0.6	200	7.1144
24.3915	0.9	300	6.5571
17.5973	1.198	400	6.1915
17.5973	1.498	500	5.9479
15.5972	1.798	600	5.7766
15.5972	2.096	700	5.6576
13.492	2.396	800	5.5389
13.492	2.6960	900	5.4367
12.6163	2.996	1000	5.3297
12.6163	3.294	1100	5.3594
10.1064	3.594	1200	5.3157
10.1064	3.894	1300	5.2836
9.3565	4.192	1400	5.3255
9.3565	4.492	1500	5.3200
8.3026	4.792	1600	5.3143

Framework versions

• Transformers 4.47.0
• Pytorch 2.5.1+cu121
• Datasets 3.3.1
• Tokenizers 0.21.0