pszemraj's picture
Update README.md
68b4958 verified
metadata
tags:
  - generated_from_trainer
metrics:
  - accuracy
license: apache-2.0
datasets:
  - pszemraj/simple_wikipedia_LM
language:
  - en

griffin-v0.01-c3t-8layer-simplewiki-silu

  • griffin/recurrent_gemma arch
  • claude3 tokenizer (as an HF gpt2 tokenizer)

Model description

pretrain experiment on the pszemraj/simple_wikipedia_LM dataset.

It achieves the following results on the evaluation set:

  • Loss: 4.0476
  • Accuracy: 0.4224

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0003
  • train_batch_size: 4
  • eval_batch_size: 4
  • seed: 80085
  • gradient_accumulation_steps: 32
  • total_train_batch_size: 128
  • optimizer: Adam with betas=(0.9,0.99) and epsilon=1e-07
  • lr_scheduler_type: constant_with_warmup
  • lr_scheduler_warmup_ratio: 0.05
  • num_epochs: 2.0

Training results

Training Loss Epoch Step Validation Loss Accuracy
13.3276 0.2548 100 12.0402 0.0131
8.9207 0.5095 200 8.0312 0.0360
7.2681 0.7643 300 6.4775 0.0506
6.3187 1.0190 400 5.6227 0.0434
5.5695 1.2738 500 4.7796 0.3635
5.2926 1.5285 600 4.3923 0.3952
4.878 1.7833 700 4.1877 0.4085

Framework versions

  • Transformers 4.40.1
  • Pytorch 2.2.0+cu121
  • Datasets 2.19.0
  • Tokenizers 0.19.1