smolm-autoreg-bpe-counterfactual_babylm_anans_new-1e-4

This model was trained from scratch on the kanishka/counterfactual_babylm_anans_new dataset. It achieves the following results on the evaluation set:

• Loss: 3.4007
• Accuracy: 0.4074

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 32
• eval_batch_size: 64
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 32000
• num_epochs: 20.0
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
4.0511	1.0	18595	4.2394	0.3093
3.5633	2.0	37190	3.7416	0.3637
3.3921	3.0	55785	3.5710	0.3807
3.2853	4.0	74380	3.5125	0.3883
3.2219	5.0	92975	3.4521	0.3931
3.1716	6.0	111570	3.4476	0.3967
3.1281	7.0	130165	3.4214	0.3994
3.0951	8.0	148760	3.4083	0.4018
3.0631	9.0	167355	3.3979	0.4024
3.0407	10.0	185950	3.3901	0.4042
3.0103	11.0	204545	3.3945	0.4041
2.9879	12.0	223140	3.3859	0.4055
2.9716	13.0	241735	3.3779	0.4063
2.9449	14.0	260330	3.3843	0.4066
2.9277	15.0	278925	3.3808	0.4074
2.9087	16.0	297520	3.3866	0.4070
2.8913	17.0	316115	3.3875	0.4071
2.8771	18.0	334710	3.3993	0.4071
2.8633	19.0	353305	3.3992	0.4073
2.8505	20.0	371900	3.4007	0.4074

Framework versions

• Transformers 4.40.1
• Pytorch 2.1.0+cu121
• Datasets 2.16.1
• Tokenizers 0.19.1