File size: 3,882 Bytes
0ea4dd5 e19dad0 0ea4dd5 e19dad0 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 |
---
language:
- zh
- en
pipeline_tag: text-generation
license: other
license_name: llama3
license_link: LICENSE
tags:
- llama3
- chinese
- meta
---
# llama-3-8b-instruct-262k-chinese-lora
llama-3-8b-instruct-262k-chinese基于[Llama-3-8B-Instruct-262k](https://huggingface.co/gradientai/Llama-3-8B-Instruct-262k)，使用ORPO方法，在中英文偏好数据集[shibing624/DPO-En-Zh-20k-Preference](https://huggingface.co/datasets/shibing624/DPO-En-Zh-20k-Preference)
上微调得到的对话模型。
模型的部署、训练等方法详见MedicalGPT的GitHub仓库：[https://github.com/shibing624/MedicalGPT](https://github.com/shibing624/MedicalGPT)
## Relate models
- 完整模型权重：https://huggingface.co/shibing624/llama-3-8b-instruct-262k-chinese
- lora权重：https://huggingface.co/shibing624/llama-3-8b-instruct-262k-chinese-lora
## 如何使用
```python
import transformers
import torch
model_id = "shibing624/llama-3-8b-instruct-262k-chinese"
pipeline = transformers.pipeline(
"text-generation",
model=model_id,
model_kwargs={"torch_dtype": torch.float16},
device="cuda",
)
messages = [{"role": "system", "content": ""}]
messages.append({"role": "user", "content": "介绍一下机器学习"})
prompt = pipeline.tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True
)
terminators = [
pipeline.tokenizer.eos_token_id,
pipeline.tokenizer.convert_tokens_to_ids("<|eot_id|>")
]
outputs = pipeline(
prompt,
max_new_tokens=512,
eos_token_id=terminators,
do_sample=True,
temperature=0.6,
top_p=0.9
)
content = outputs[0]["generated_text"][len(prompt):]
print(content)
```
## About Llama-3-8B-Instruct-262k
Gradient incorporates your data to deploy autonomous assistants that power critical operations across your business. To learn more or collaborate on a custom model.
This model extends LLama-3 8B's context length from 8k to -> 160K, developed by Gradient, sponsored by compute from [Crusoe Energy](https://huggingface.co/crusoeai). It demonstrates that SOTA LLMs can learn to operate on long context with minimal training (< 200M tokens) by appropriately adjusting RoPE theta.
<img src="https://cdn-uploads.huggingface.co/production/uploads/6585dc9be92bc5f258156bd6/hiHWva3CbsrnPvZTp5-lu.png" width="600">
**Approach:**
- [meta-llama/Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) as the base
- NTK-aware interpolation [1] to initialize an optimal schedule for RoPE theta, followed by a new data-driven RoPE theta optimization technique
- Progressive training on increasing context lengths similar to the [Large World Model](https://huggingface.co/LargeWorldModel) [2] (See details below)
**Infra:**
We build on top of the EasyContext Blockwise RingAttention library [3] to scalably and efficiently train on contexts up to 262144 tokens on [Crusoe Energy](https://huggingface.co/crusoeai) high performance L40S cluster.
**Data:**
For training data, we generate long contexts by augmenting [SlimPajama](https://huggingface.co/datasets/cerebras/SlimPajama-627B).
**Progressive Training Details:**
| Parameter | 65K | 262K |
|-----------------------------|----------------|------------|
| Initialize From | LLaMA-3-8B-Inst| 65K |
| Sequence Length | 2^16 | 2^18 |
| RoPE theta | 15.3 M | 207.1 M |
| Batch Size (Tokens / Step) | 2.097 M | 4.192 M |
| Steps | 30 | 24 |
| Total Tokens | 63 M | 101 M |
| Learning Rate | 2.00E-05 | 2.00E-05 |
| # GPUs | 32 | 32 |
| GPU Type | NVIDIA L40S | NVIDIA L40S|
|