metadata

tags:
  - generated_from_trainer
datasets:
  - kanishka/counterfactual-babylm-only_other_det_removal
metrics:
  - accuracy
model-index:
  - name: >-
      smolm-autoreg-bpe-counterfactual-babylm-only_other_det_removal-seed_211-1e-3
    results:
      - task:
          name: Causal Language Modeling
          type: text-generation
        dataset:
          name: kanishka/counterfactual-babylm-only_other_det_removal
          type: kanishka/counterfactual-babylm-only_other_det_removal
        metrics:
          - name: Accuracy
            type: accuracy
            value: 0.4109943845202858

smolm-autoreg-bpe-counterfactual-babylm-only_other_det_removal-seed_211-1e-3

This model was trained from scratch on the kanishka/counterfactual-babylm-only_other_det_removal dataset. It achieves the following results on the evaluation set:

Loss: 3.4143
Accuracy: 0.4110

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 0.001
train_batch_size: 32
eval_batch_size: 64
seed: 211
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: linear
lr_scheduler_warmup_steps: 32000
num_epochs: 20.0
mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
3.6004	1.0	18597	3.8219	0.3575
3.3852	2.0	37194	3.6092	0.3797
3.2597	3.0	55791	3.4837	0.3910
3.1758	4.0	74388	3.4364	0.3981
3.1197	5.0	92985	3.4116	0.4017
3.08	6.0	111582	3.3782	0.4040
3.0418	7.0	130179	3.3885	0.4055
3.0088	8.0	148776	3.3884	0.4062
2.9856	9.0	167373	3.3548	0.4077
2.9598	10.0	185970	3.3782	0.4090
2.9364	11.0	204567	3.3851	0.4093
2.9156	12.0	223164	3.3803	0.4097
2.8949	13.0	241761	3.3869	0.4100
2.8719	14.0	260358	3.3813	0.4104
2.8526	15.0	278955	3.3859	0.4108
2.8289	16.0	297552	3.3980	0.4103
2.8104	17.0	316149	3.3981	0.4109
2.7958	18.0	334746	3.4054	0.4110
2.781	19.0	353343	3.4057	0.4110
2.7571	20.0	371940	3.4143	0.4110

Framework versions

Transformers 4.37.2
Pytorch 2.1.0+cu121
Datasets 2.16.1
Tokenizers 0.15.1