smolm-autoreg-bpe-counterfactual-babylm-new_regex_aanns_removal-seed_211-1e-3

This model was trained from scratch on the kanishka/counterfactual-babylm-new_regex_aanns_removal dataset. It achieves the following results on the evaluation set:

• Loss: 3.3885
• Accuracy: 0.4134

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.001
• train_batch_size: 32
• eval_batch_size: 64
• seed: 211
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 32000
• num_epochs: 20.0
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
3.6058	1.0	18593	3.7591	0.3583
3.3838	2.0	37186	3.5626	0.3824
3.2578	3.0	55779	3.4700	0.3923
3.1777	4.0	74372	3.4090	0.3992
3.1262	5.0	92965	3.4092	0.4021
3.0786	6.0	111558	3.3686	0.4073
3.0425	7.0	130151	3.3363	0.4099
3.0098	8.0	148744	3.3507	0.4092
2.9845	9.0	167337	3.3483	0.4113
2.9554	10.0	185930	3.3369	0.4122
2.9372	11.0	204523	3.3210	0.4144
2.9131	12.0	223116	3.3488	0.4121
2.8914	13.0	241709	3.3448	0.4139
2.8744	14.0	260302	3.3473	0.4130
2.8505	15.0	278895	3.3552	0.4135
2.8346	16.0	297488	3.3626	0.4135
2.8113	17.0	316081	3.3734	0.4128
2.7967	18.0	334674	3.3720	0.4132
2.7775	19.0	353267	3.3848	0.4132
2.7551	20.0	371860	3.3885	0.4134

Framework versions

• Transformers 4.38.0
• Pytorch 2.3.1+cu121
• Datasets 2.16.1
• Tokenizers 0.15.2