collapse_gemma-2-2b_hs2_iter1_sftsd0

This model is a fine-tuned version of google/gemma-2-2b on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.0645
• Num Input Tokens Seen: 5698680

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 8e-06
• train_batch_size: 8
• eval_batch_size: 16
• seed: 0
• gradient_accumulation_steps: 16
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant_with_warmup
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
No log	0	0	1.3956	0
1.3437	0.0511	5	1.2632	296352
1.1851	0.1021	10	1.1721	589152
1.1271	0.1532	15	1.1344	884504
1.0728	0.2042	20	1.1085	1182424
1.0945	0.2553	25	1.0987	1474984
1.0927	0.3063	30	1.0931	1772592
1.1145	0.3574	35	1.0890	2061504
1.0845	0.4084	40	1.0854	2358064
1.1001	0.4595	45	1.0824	2650896
1.0775	0.5105	50	1.0801	2942864
1.1246	0.5616	55	1.0775	3234512
1.101	0.6126	60	1.0753	3525376
1.0904	0.6637	65	1.0739	3820376
1.1705	0.7147	70	1.0718	4108240
1.0282	0.7658	75	1.0702	4402208
1.1463	0.8168	80	1.0689	4698016
1.0783	0.8679	85	1.0675	4991408
1.0052	0.9190	90	1.0657	5285784
1.0614	0.9700	95	1.0648	5580576

Framework versions

• Transformers 4.44.0
• Pytorch 2.4.0+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1