gradientai
/

Llama-3-8B-Instruct-262k

Text Generation

text-generation-inference

Inference Endpoints

Model card Files Files and versions Community

michaelfeil commited on Apr 25

Commit

3d44796

•

1 Parent(s): 990790f

Update README.md

Files changed (1) hide show

README.md +1 -1

README.md CHANGED Viewed

@@ -35,7 +35,7 @@ For training data, we generate long contexts by augmenting [SlimPajama](https://
 | Initialize From             | LLaMA-3-8B-Inst| 65K        |
 | Sequence Length             | 2^16           | 2^18       |
 | RoPE theta                  | 15.3 M         | 207.1 M    |
-| Batch Size (Tokens / Step)  | 2M             | 4M         |
 | Steps                       | 30             | 24         |
 | Total Tokens                | 63 M           | 101 M      |
 | Learning Rate               | 2.00E-05       | 2.00E-05   |

 | Initialize From             | LLaMA-3-8B-Inst| 65K        |
 | Sequence Length             | 2^16           | 2^18       |
 | RoPE theta                  | 15.3 M         | 207.1 M    |
+| Batch Size (Tokens / Step)  | 2.097 M        | 4.192 M    |
 | Steps                       | 30             | 24         |
 | Total Tokens                | 63 M           | 101 M      |
 | Learning Rate               | 2.00E-05       | 2.00E-05   |