smolm-autoreg-bpe-babylm-seed_211-1e-3

This model was trained from scratch on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 3.4078
• Accuracy: 0.4111

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.001
• train_batch_size: 32
• eval_batch_size: 128
• seed: 211
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 32000
• num_epochs: 20.0
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
3.6026	1.0	18595	3.7870	0.3578
3.3834	2.0	37190	3.5755	0.3795
3.2539	3.0	55785	3.4650	0.3916
3.1778	4.0	74380	3.4013	0.3993
3.1219	5.0	92975	3.3872	0.4012
3.0829	6.0	111570	3.3593	0.4048
3.0448	7.0	130165	3.3473	0.4077
3.0113	8.0	148760	3.3620	0.4074
2.9876	9.0	167355	3.3488	0.4084
2.961	10.0	185950	3.3557	0.4087
2.9378	11.0	204545	3.3599	0.4096
2.9178	12.0	223140	3.3431	0.4103
2.8924	13.0	241735	3.3626	0.4109
2.8737	14.0	260330	3.3569	0.4114
2.8492	15.0	278925	3.3599	0.4112
2.8356	16.0	297520	3.3829	0.4106
2.8162	17.0	316115	3.3819	0.4111
2.7966	18.0	334710	3.3926	0.4110
2.7802	19.0	353305	3.3977	0.4115
2.7617	20.0	371900	3.4078	0.4111

Framework versions

• Transformers 4.35.0
• Pytorch 2.1.0+cu121
• Datasets 2.12.0
• Tokenizers 0.14.1