Commit
9e8091a
1 Parent(s): 8bb6d18

Update README.md (#15)

Browse files

- Update README.md (94f7ba2d76c1644875a50aedcb774cf16d3ef0ca)


Co-authored-by: Leo Pekelis <leo-pekelis-gradient@users.noreply.huggingface.co>

Files changed (1) hide show
  1. README.md +7 -2
README.md CHANGED
@@ -16,7 +16,9 @@ For more info see our [End-to-end development service for custom LLMs and AI sys
16
 
17
  This model extends LLama-3 8B's context length from 8k to > 1040K, developed by Gradient, sponsored by compute from [Crusoe Energy](https://huggingface.co/crusoeai). It demonstrates that SOTA LLMs can learn to operate on long context with minimal training by appropriately adjusting RoPE theta. We trained on 830M tokens for this stage, and 1.4B tokens total for all stages, which is < 0.01% of Llama-3's original pre-training data.
18
 
19
- ![image/png](https://cdn-uploads.huggingface.co/production/uploads/6585dc9be92bc5f258156bd6/6MKLoX2ruLIaREiyb6coO.png)
 
 
20
 
21
  **Approach:**
22
 
@@ -32,7 +34,7 @@ Notably, we layered parallelism on top of Ring Attention with a custom network t
32
 
33
  **Data:**
34
 
35
- For training data, we generate long contexts by augmenting [SlimPajama](https://huggingface.co/datasets/cerebras/SlimPajama-627B).
36
 
37
  **Progressive Training Details:**
38
 
@@ -100,6 +102,9 @@ Drop an email to [contact@gradient.ai](mailto:contact@gradient.ai)
100
 
101
  [3] https://github.com/jzhang38/EasyContext
102
 
 
 
 
103
 
104
  ----
105
 
 
16
 
17
  This model extends LLama-3 8B's context length from 8k to > 1040K, developed by Gradient, sponsored by compute from [Crusoe Energy](https://huggingface.co/crusoeai). It demonstrates that SOTA LLMs can learn to operate on long context with minimal training by appropriately adjusting RoPE theta. We trained on 830M tokens for this stage, and 1.4B tokens total for all stages, which is < 0.01% of Llama-3's original pre-training data.
18
 
19
+ **Update (5/3): We further fine-tuned our model to strengthen its assistant-like chat ability as well. The NIAH result is updated.**
20
+
21
+ ![image/png](https://cdn-uploads.huggingface.co/production/uploads/6585dc9be92bc5f258156bd6/-qaI__83ksClzoJzlqZjq.png)
22
 
23
  **Approach:**
24
 
 
34
 
35
  **Data:**
36
 
37
+ For training data, we generate long contexts by augmenting [SlimPajama](https://huggingface.co/datasets/cerebras/SlimPajama-627B). We also fine-tune on a chat dataset based on UltraChat [4], following a similar recipe for data augmentation to [2].
38
 
39
  **Progressive Training Details:**
40
 
 
102
 
103
  [3] https://github.com/jzhang38/EasyContext
104
 
105
+ [3] Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan
106
+ Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling
107
+ high-quality instructional conversations. arXiv preprint arXiv:2305.14233, 2023.
108
 
109
  ----
110