kanishka's picture
Model save
3cbf124 verified
metadata
tags:
  - generated_from_trainer
metrics:
  - accuracy
model-index:
  - name: smolm-autoreg-bpe-counterfactual-babylm-pipps_removal-seed_211-1e-3
    results: []

smolm-autoreg-bpe-counterfactual-babylm-pipps_removal-seed_211-1e-3

This model was trained from scratch on an unknown dataset. It achieves the following results on the evaluation set:

  • Loss: 3.4007
  • Accuracy: 0.4100

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.001
  • train_batch_size: 32
  • eval_batch_size: 64
  • seed: 211
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • lr_scheduler_warmup_steps: 32000
  • num_epochs: 20.0
  • mixed_precision_training: Native AMP

Training results

Training Loss Epoch Step Validation Loss Accuracy
3.6079 1.0 18593 3.8149 0.3576
3.3841 2.0 37186 3.5902 0.3792
3.2548 3.0 55779 3.4807 0.3918
3.1845 4.0 74372 3.4469 0.3968
3.1246 5.0 92965 3.4169 0.4014
3.0823 6.0 111558 3.3873 0.4035
3.0457 7.0 130151 3.3857 0.4053
3.0112 8.0 148744 3.3520 0.4070
2.9878 9.0 167337 3.3733 0.4072
2.96 10.0 185930 3.3503 0.4083
2.938 11.0 204523 3.3664 0.4084
2.9158 12.0 223116 3.3660 0.4093
2.8919 13.0 241709 3.3564 0.4101
2.8735 14.0 260302 3.3567 0.4107
2.8562 15.0 278895 3.3675 0.4100
2.8344 16.0 297488 3.3702 0.4103
2.814 17.0 316081 3.3808 0.4101
2.7973 18.0 334674 3.3935 0.4098
2.7732 19.0 353267 3.3887 0.4104
2.7585 20.0 371860 3.4007 0.4100

Framework versions

  • Transformers 4.37.2
  • Pytorch 2.1.0+cu121
  • Datasets 2.16.1
  • Tokenizers 0.15.1