smolm-autoreg-bpe-counterfactual-babylm-pipps_removal-seed_211-1e-3

This model was trained from scratch on the kanishka/counterfactual-babylm-pipps_removal dataset. It achieves the following results on the evaluation set:

• Loss: 3.4007
• Accuracy: 0.4100

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.001
• train_batch_size: 32
• eval_batch_size: 64
• seed: 211
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 32000
• num_epochs: 20.0
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
3.6079	1.0	18593	3.8149	0.3576
3.3841	2.0	37186	3.5902	0.3792
3.2548	3.0	55779	3.4807	0.3918
3.1845	4.0	74372	3.4469	0.3968
3.1246	5.0	92965	3.4169	0.4014
3.0823	6.0	111558	3.3873	0.4035
3.0457	7.0	130151	3.3857	0.4053
3.0112	8.0	148744	3.3520	0.4070
2.9878	9.0	167337	3.3733	0.4072
2.96	10.0	185930	3.3503	0.4083
2.938	11.0	204523	3.3664	0.4084
2.9158	12.0	223116	3.3660	0.4093
2.8919	13.0	241709	3.3564	0.4101
2.8735	14.0	260302	3.3567	0.4107
2.8562	15.0	278895	3.3675	0.4100
2.8344	16.0	297488	3.3702	0.4103
2.814	17.0	316081	3.3808	0.4101
2.7973	18.0	334674	3.3935	0.4098
2.7732	19.0	353267	3.3887	0.4104
2.7585	20.0	371860	3.4007	0.4100

Framework versions

• Transformers 4.37.2
• Pytorch 2.1.0+cu121
• Datasets 2.16.1
• Tokenizers 0.15.1