Model save

README.md

@@ -0,0 +1,78 @@
+---
+license: apache-2.0
+base_model: pszemraj/griffin-v0.01-c3t-8layer-simplewiki-silu
+tags:
+- generated_from_trainer
+metrics:
+- accuracy
+model-index:
+- name: griffin-v0.01-c3t-8layer-simplewiki-silu-fineweb-1M_en-med-vN
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+# griffin-v0.01-c3t-8layer-simplewiki-silu-fineweb-1M_en-med-vN
+
+This model is a fine-tuned version of [pszemraj/griffin-v0.01-c3t-8layer-simplewiki-silu](https://huggingface.co/pszemraj/griffin-v0.01-c3t-8layer-simplewiki-silu) on an unknown dataset.
+It achieves the following results on the evaluation set:
+- Loss: 5.1981
+- Accuracy: 0.2315
+- Num Input Tokens Seen: 786432000
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 0.0003
+- train_batch_size: 2
+- eval_batch_size: 2
+- seed: 80085
+- gradient_accumulation_steps: 32
+- total_train_batch_size: 64
+- optimizer: Adam with betas=(0.9,0.99) and epsilon=1e-07
+- lr_scheduler_type: inverse_sqrt
+- lr_scheduler_warmup_ratio: 0.05
+- num_epochs: 1.0
+
+### Training results
+
+| Training Loss | Epoch  | Step | Validation Loss | Accuracy | Input Tokens Seen |
+|:-------------:|:------:|:----:|:---------------:|:--------:|:-----------------:|
+| 6.0703        | 0.0656 | 400  | 6.2332          | 0.1701   | 52428800          |
+| 5.723         | 0.1313 | 800  | 5.9116          | 0.1893   | 104857600         |
+| 5.5106        | 0.1969 | 1200 | 5.7516          | 0.1976   | 157286400         |
+| 5.455         | 0.2626 | 1600 | 5.6427          | 0.2032   | 209715200         |
+| 5.3236        | 0.3282 | 2000 | 5.5567          | 0.2103   | 262144000         |
+| 5.2764        | 0.3938 | 2400 | 5.4919          | 0.2151   | 314572800         |
+| 5.1625        | 0.4595 | 2800 | 5.4436          | 0.2176   | 367001600         |
+| 5.1851        | 0.5251 | 3200 | 5.3975          | 0.2206   | 419430400         |
+| 5.0618        | 0.5908 | 3600 | 5.3624          | 0.2199   | 471859200         |
+| 5.0278        | 0.6564 | 4000 | 5.3242          | 0.2236   | 524288000         |
+| 5.0389        | 0.7220 | 4400 | 5.2920          | 0.2264   | 576716800         |
+| 4.9732        | 0.7877 | 4800 | 5.2674          | 0.2276   | 629145600         |
+| 4.9375        | 0.8533 | 5200 | 5.2418          | 0.2292   | 681574400         |
+| 4.9322        | 0.9190 | 5600 | 5.2166          | 0.2312   | 734003200         |
+| 4.8818        | 0.9846 | 6000 | 5.1981          | 0.2315   | 786432000         |
+
+
+### Framework versions
+
+- Transformers 4.40.1
+- Pytorch 2.3.0+cu121
+- Datasets 2.19.0
+- Tokenizers 0.19.1

generation_config.json

@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "eos_token_id": 0,
+  "pad_token_id": 0,
+  "transformers_version": "4.40.1"
+}

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:927e37a5a04757a250a3824d6d2e92f9b5bb028ed781002058ce33be1233898a
+oid sha256:a8f29f7b9698de4eec4ee6a7d9585712e3abea33c4545e11ef52e6022f3602ef
 size 671684224