leo-pekelis-gradient
commited on
Commit
•
eadf5aa
1
Parent(s):
1c075c4
Update README.md
Browse files
README.md
CHANGED
@@ -50,7 +50,31 @@ For training data, we generate long contexts by augmenting [SlimPajama](https://
|
|
50 |
| GPU Type | NVIDIA L40S | NVIDIA L40S | NVIDIA L40S | NVIDIA L40S |
|
51 |
| Minutes to Train (Wall)| 202 | 555 | 61 | 87 |
|
52 |
|
53 |
-
**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
54 |
- [GGUF](https://huggingface.co/crusoeai/Llama-3-8B-Instruct-1048k-GGUF)
|
55 |
- [MLX-4bit](https://huggingface.co/mlx-community/Llama-3-8B-Instruct-1048k-4bit)
|
56 |
|
|
|
50 |
| GPU Type | NVIDIA L40S | NVIDIA L40S | NVIDIA L40S | NVIDIA L40S |
|
51 |
| Minutes to Train (Wall)| 202 | 555 | 61 | 87 |
|
52 |
|
53 |
+
**Evaluation:**
|
54 |
+
|
55 |
+
![image/png](https://cdn-uploads.huggingface.co/production/uploads/6585dc9be92bc5f258156bd6/mWxIGZNi3ejlmeIDWafKu.png)
|
56 |
+
|
57 |
+
```
|
58 |
+
EVAL_MAX_CONTEXT_LENGTH=1040200
|
59 |
+
EVAL_MIN_CONTEXT_LENGTH=100
|
60 |
+
EVAL_CONTEXT_INTERVAL=86675
|
61 |
+
EVAL_DEPTH_INTERVAL=0.2
|
62 |
+
EVAL_RND_NUMBER_DIGITS=8
|
63 |
+
|
64 |
+
HAYSTACK1:
|
65 |
+
EVAL_GENERATOR_TOKENS=25
|
66 |
+
|
67 |
+
HAYSTACK2:
|
68 |
+
EVAL_CONTEXT_INTERVAL=173350
|
69 |
+
EVAL_GENERATOR_TOKENS=150000
|
70 |
+
|
71 |
+
HAYSTACK3:
|
72 |
+
EVAL_GENERATOR_TOKENS=925000
|
73 |
+
```
|
74 |
+
|
75 |
+
All boxes not pictured for Haystack 1 and 3 are 100% accurate. Haystacks 1,2 and 3 are further detailed in this [blog post](https://gradient.ai/blog/the-haystack-matters-for-niah-evals).
|
76 |
+
|
77 |
+
**Quants:**
|
78 |
- [GGUF](https://huggingface.co/crusoeai/Llama-3-8B-Instruct-1048k-GGUF)
|
79 |
- [MLX-4bit](https://huggingface.co/mlx-community/Llama-3-8B-Instruct-1048k-4bit)
|
80 |
|