gradientai
/

Llama-3-70B-Instruct-Gradient-1048k

Text Generation

Inference Endpoints

text-generation-inference

Model card Files Files and versions Community

tpeng726 commited on May 4

Commit

f993ebb

•

1 Parent(s): 4cf2015

Update eval plot

Files changed (1) hide show

README.md +1 -1

README.md CHANGED Viewed

@@ -17,7 +17,7 @@ For more info see our [End-to-end development service for custom LLMs and AI sys
 This model extends LLama-3 70B's context length from 8k to > 1048K, developed by Gradient, sponsored by compute from [Crusoe Energy](https://huggingface.co/crusoeai). It demonstrates that SOTA LLMs can learn to operate on long context with minimal training by appropriately adjusting RoPE theta. We trained on 34M tokens for this stage, and ~430M tokens total for all stages, which is < 0.003% of Llama-3's original pre-training data.
-![image/png](https://cdn-uploads.huggingface.co/production/uploads/644fac0ce1d7a97f3b653ab1/GLrUQLqji6qFu_pxpBW7O.png)
 **Approach:**

 This model extends LLama-3 70B's context length from 8k to > 1048K, developed by Gradient, sponsored by compute from [Crusoe Energy](https://huggingface.co/crusoeai). It demonstrates that SOTA LLMs can learn to operate on long context with minimal training by appropriately adjusting RoPE theta. We trained on 34M tokens for this stage, and ~430M tokens total for all stages, which is < 0.003% of Llama-3's original pre-training data.
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/655bb613e8a8971e89944f3e/gb6Z7nMQR5WoLUsW35CIh.png)
 **Approach:**