kanishka's picture
Model save
ae396fd verified
metadata
tags:
  - generated_from_trainer
metrics:
  - accuracy
model-index:
  - name: >-
      smolm-autoreg-bpe-counterfactual-babylm-only_measure_nps_as_singular_removal-seed_211-1e-3
    results: []

smolm-autoreg-bpe-counterfactual-babylm-only_measure_nps_as_singular_removal-seed_211-1e-3

This model was trained from scratch on an unknown dataset. It achieves the following results on the evaluation set:

  • Loss: 3.4372
  • Accuracy: 0.4092

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.001
  • train_batch_size: 32
  • eval_batch_size: 64
  • seed: 211
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • lr_scheduler_warmup_steps: 32000
  • num_epochs: 20.0
  • mixed_precision_training: Native AMP

Training results

Training Loss Epoch Step Validation Loss Accuracy
3.6018 1.0 18600 3.7779 0.3590
3.3799 2.0 37200 3.5990 0.3799
3.2535 3.0 55800 3.4629 0.3928
3.1731 4.0 74400 3.4447 0.3979
3.1186 5.0 93000 3.4295 0.4009
3.0776 6.0 111600 3.4004 0.4034
3.0407 7.0 130200 3.3850 0.4053
3.0066 8.0 148800 3.3648 0.4061
2.9851 9.0 167400 3.3985 0.4074
2.953 10.0 186000 3.3964 0.4077
2.9321 11.0 204600 3.3816 0.4088
2.9082 12.0 223200 3.3780 0.4093
2.8881 13.0 241800 3.4020 0.4090
2.8698 14.0 260400 3.4057 0.4091
2.8441 15.0 279000 3.3906 0.4094
2.8256 16.0 297600 3.4051 0.4094
2.808 17.0 316200 3.4108 0.4093
2.7945 18.0 334800 3.4283 0.4094
2.7744 19.0 353400 3.4362 0.4094
2.7567 20.0 372000 3.4372 0.4092

Framework versions

  • Transformers 4.37.2
  • Pytorch 2.1.0+cu121
  • Datasets 2.16.1
  • Tokenizers 0.15.1