---
license: apache-2.0
base_model: pszemraj/griffin-v0.01-c3t-8layer-simplewiki-silu
tags:
- generated_from_trainer
metrics:
- accuracy
model-index:
- name: griffin-v0.01-c3t-8layer-simplewiki-silu-fineweb-1M_en-med-vN
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# griffin-v0.01-c3t-8layer-simplewiki-silu-fineweb-1M_en-med-vN

This model is a fine-tuned version of [pszemraj/griffin-v0.01-c3t-8layer-simplewiki-silu](https://huggingface.co/pszemraj/griffin-v0.01-c3t-8layer-simplewiki-silu) on the BEE-spoke-data/fineweb-1M_en-med dataset.
It achieves the following results on the evaluation set:
- Loss: 5.1888
- Accuracy: 0.2326
- Num Input Tokens Seen: 798621696

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0003
- train_batch_size: 2
- eval_batch_size: 2
- seed: 80085
- gradient_accumulation_steps: 32
- total_train_batch_size: 64
- optimizer: Adam with betas=(0.9,0.99) and epsilon=1e-07
- lr_scheduler_type: inverse_sqrt
- lr_scheduler_warmup_ratio: 0.05
- num_epochs: 1.0

### Training results

| Training Loss | Epoch  | Step | Validation Loss | Accuracy | Input Tokens Seen |
|:-------------:|:------:|:----:|:---------------:|:--------:|:-----------------:|
| 6.0703        | 0.0656 | 400  | 6.2332          | 0.1701   | 52428800          |
| 5.723         | 0.1313 | 800  | 5.9116          | 0.1893   | 104857600         |
| 5.5106        | 0.1969 | 1200 | 5.7516          | 0.1976   | 157286400         |
| 5.455         | 0.2626 | 1600 | 5.6427          | 0.2032   | 209715200         |
| 5.3236        | 0.3282 | 2000 | 5.5567          | 0.2103   | 262144000         |
| 5.2764        | 0.3938 | 2400 | 5.4919          | 0.2151   | 314572800         |
| 5.1625        | 0.4595 | 2800 | 5.4436          | 0.2176   | 367001600         |
| 5.1851        | 0.5251 | 3200 | 5.3975          | 0.2206   | 419430400         |
| 5.0618        | 0.5908 | 3600 | 5.3624          | 0.2199   | 471859200         |
| 5.0278        | 0.6564 | 4000 | 5.3242          | 0.2236   | 524288000         |
| 5.0389        | 0.7220 | 4400 | 5.2920          | 0.2264   | 576716800         |
| 4.9732        | 0.7877 | 4800 | 5.2674          | 0.2276   | 629145600         |
| 4.9375        | 0.8533 | 5200 | 5.2418          | 0.2292   | 681574400         |
| 4.9322        | 0.9190 | 5600 | 5.2166          | 0.2312   | 734003200         |
| 4.8818        | 0.9846 | 6000 | 5.1981          | 0.2315   | 786432000         |


### Framework versions

- Transformers 4.40.1
- Pytorch 2.3.0+cu121
- Datasets 2.19.0
- Tokenizers 0.19.1