Update README.md

README.md

@@ -1,3 +1,102 @@
 ---
-license: llama3
+language:
+- zh
+- en
+pipeline_tag: text-generation
+license: other
+license_name: llama3
+license_link: LICENSE
+tags:
+- llama3
+- chinese
+- meta
 ---
+
+
+# llama-3-8b-instruct-262k-chinese-lora
+
+
+llama-3-8b-instruct-262k-chinese基于[Llama-3-8B-Instruct-262k](https://huggingface.co/gradientai/Llama-3-8B-Instruct-262k)，使用ORPO方法，在中英文偏好数据集[shibing624/DPO-En-Zh-20k-Preference](https://huggingface.co/datasets/shibing624/DPO-En-Zh-20k-Preference)
+上微调得到的对话模型。
+
+模型的部署、训练等方法详见MedicalGPT的GitHub仓库：[https://github.com/shibing624/MedicalGPT](https://github.com/shibing624/MedicalGPT)
+## Relate models
+- 完整模型权重：https://huggingface.co/shibing624/llama-3-8b-instruct-262k-chinese
+- lora权重：https://huggingface.co/shibing624/llama-3-8b-instruct-262k-chinese-lora
+## 如何使用
+
+```python
+import transformers
+import torch
+
+model_id = "shibing624/llama-3-8b-instruct-262k-chinese"
+pipeline = transformers.pipeline(
+    "text-generation",
+    model=model_id,
+    model_kwargs={"torch_dtype": torch.float16},
+    device="cuda",
+)
+
+messages = [{"role": "system", "content": ""}]
+messages.append({"role": "user", "content": "介绍一下机器学习"})
+prompt = pipeline.tokenizer.apply_chat_template(
+        messages, 
+        tokenize=False, 
+        add_generation_prompt=True
+    )
+terminators = [
+        pipeline.tokenizer.eos_token_id,
+        pipeline.tokenizer.convert_tokens_to_ids("<|eot_id|>")
+    ]
+outputs = pipeline(
+    prompt,
+    max_new_tokens=512,
+    eos_token_id=terminators,
+    do_sample=True,
+    temperature=0.6,
+    top_p=0.9
+)
+content = outputs[0]["generated_text"][len(prompt):]
+print(content)
+```
+
+
+## About Llama-3-8B-Instruct-262k
+Gradient incorporates your data to deploy autonomous assistants that power critical operations across your business. To learn more or collaborate on a custom model.
+
+This model extends LLama-3 8B's context length from 8k to -> 160K, developed by Gradient, sponsored by compute from [Crusoe Energy](https://huggingface.co/crusoeai). It demonstrates that SOTA LLMs can learn to operate on long context with minimal training (< 200M tokens) by appropriately adjusting RoPE theta. 
+
+<img src="https://cdn-uploads.huggingface.co/production/uploads/6585dc9be92bc5f258156bd6/hiHWva3CbsrnPvZTp5-lu.png" width="600">
+
+**Approach:**
+
+- [meta-llama/Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) as the base
+- NTK-aware interpolation [1] to initialize an optimal schedule for RoPE theta, followed by a new data-driven RoPE theta optimization technique
+- Progressive training on increasing context lengths similar to the [Large World Model](https://huggingface.co/LargeWorldModel) [2] (See details below)
+
+**Infra:**
+
+We build on top of the EasyContext Blockwise RingAttention library [3] to scalably and efficiently train on contexts up to 262144 tokens on [Crusoe Energy](https://huggingface.co/crusoeai) high performance L40S cluster. 
+
+**Data:**
+
+For training data, we generate long contexts by augmenting [SlimPajama](https://huggingface.co/datasets/cerebras/SlimPajama-627B).
+
+**Progressive Training Details:**
+
+| Parameter                   | 65K            | 262K       |
+|-----------------------------|----------------|------------|
+| Initialize From             | LLaMA-3-8B-Inst| 65K        |
+| Sequence Length             | 2^16           | 2^18       |
+| RoPE theta                  | 15.3 M         | 207.1 M    |
+| Batch Size (Tokens / Step)  | 2.097 M        | 4.192 M    |
+| Steps                       | 30             | 24         |
+| Total Tokens                | 63 M           | 101 M      |
+| Learning Rate               | 2.00E-05       | 2.00E-05   |
+| # GPUs                      | 32             | 32         |
+| GPU Type                    | NVIDIA L40S    | NVIDIA L40S|
+
+
+
+
+