opt-babylm2-rewritten-clean-spacy_random-removal-num-adj-earlystop-bpe_seed-211_1e-3

This model was trained from scratch on the kanishka/babylm2-rewritten-clean-spacy_random-removal-num-adj dataset. It achieves the following results on the evaluation set:

• Loss: 2.6845
• Accuracy: 0.4788

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.001
• train_batch_size: 32
• eval_batch_size: 64
• seed: 211
• gradient_accumulation_steps: 8
• total_train_batch_size: 256
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 32000
• num_epochs: 20.0
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
4.1014	0.9997	2240	3.8208	0.3603
3.4582	1.9997	4480	3.3074	0.4088
3.1363	2.9997	6720	3.0891	0.4308
2.9666	3.9997	8960	2.9839	0.4412
2.844	4.9997	11200	2.9235	0.4477
2.788	5.9997	13440	2.8839	0.4515
2.7451	6.9997	15680	2.8566	0.4546
2.7089	7.9997	17920	2.8379	0.4566
2.6905	8.9997	20160	2.8257	0.4583
2.6709	9.9997	22400	2.8130	0.4599
2.6545	10.9997	24640	2.8054	0.4606
2.6375	11.9997	26880	2.7972	0.4615
2.6417	12.9997	29120	2.7929	0.4622
2.6319	13.9997	31360	2.7897	0.4625
2.6174	14.9997	33600	2.7694	0.4652
2.5717	15.9997	35840	2.7437	0.4686
2.5216	16.9997	38080	2.7195	0.4717
2.4671	17.9997	40320	2.7021	0.4746
2.4024	18.9997	42560	2.6872	0.4773
2.3298	19.9997	44800	2.6845	0.4788

Framework versions

• Transformers 4.48.0
• Pytorch 2.5.1
• Datasets 3.2.0
• Tokenizers 0.21.0