kanishka's picture
Model save
5776cdf
metadata
tags:
  - generated_from_trainer
metrics:
  - accuracy
model-index:
  - name: smolm-autoreg-bpe-counterfactual-babylm-indef-removal-seed_1024-1e-4
    results: []

smolm-autoreg-bpe-counterfactual-babylm-indef-removal-seed_1024-1e-4

This model was trained from scratch on the None dataset. It achieves the following results on the evaluation set:

  • Loss: 3.4350
  • Accuracy: 0.4063

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0001
  • train_batch_size: 32
  • eval_batch_size: 64
  • seed: 1024
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • lr_scheduler_warmup_steps: 32000
  • num_epochs: 20.0
  • mixed_precision_training: Native AMP

Training results

Training Loss Epoch Step Validation Loss Accuracy
4.0569 1.0 18595 4.2925 0.3078
3.5613 2.0 37190 3.7467 0.3633
3.3894 3.0 55785 3.5963 0.3802
3.2856 4.0 74380 3.5303 0.3880
3.2222 5.0 92975 3.4793 0.3935
3.1693 6.0 111570 3.4552 0.3967
3.13 7.0 130165 3.4350 0.3992
3.0982 8.0 148760 3.4201 0.4011
3.0659 9.0 167355 3.4224 0.4025
3.0359 10.0 185950 3.4172 0.4028
3.0157 11.0 204545 3.4154 0.4036
2.9936 12.0 223140 3.4075 0.4043
2.9676 13.0 241735 3.4182 0.4048
2.9444 14.0 260330 3.4119 0.4057
2.9265 15.0 278925 3.4089 0.4056
2.9105 16.0 297520 3.4230 0.4058
2.8892 17.0 316115 3.4260 0.4060
2.8774 18.0 334710 3.4290 0.4063
2.8592 19.0 353305 3.4288 0.4065
2.8472 20.0 371900 3.4350 0.4063

Framework versions

  • Transformers 4.35.0
  • Pytorch 2.1.0+cu121
  • Datasets 2.12.0
  • Tokenizers 0.14.1