metadata

language:
  - zh
  - en
pipeline_tag: text-generation
license: other
license_name: llama3
license_link: LICENSE
tags:
  - llama3
  - chinese
  - meta

llama-3-8b-instruct-262k-chinese-lora

llama-3-8b-instruct-262k-chinese基于Llama-3-8B-Instruct-262k，使用ORPO方法，在中英文偏好数据集shibing624/DPO-En-Zh-20k-Preference 上微调得到的对话模型。

模型的部署、训练等方法详见MedicalGPT的GitHub仓库：https://github.com/shibing624/MedicalGPT

Relate models

完整模型权重：https://huggingface.co/shibing624/llama-3-8b-instruct-262k-chinese
lora权重：https://huggingface.co/shibing624/llama-3-8b-instruct-262k-chinese-lora

如何使用

import transformers
import torch

model_id = "shibing624/llama-3-8b-instruct-262k-chinese"
pipeline = transformers.pipeline(
    "text-generation",
    model=model_id,
    model_kwargs={"torch_dtype": torch.float16},
    device="cuda",
)

messages = [{"role": "system", "content": ""}]
messages.append({"role": "user", "content": "介绍一下机器学习"})
prompt = pipeline.tokenizer.apply_chat_template(
        messages, 
        tokenize=False, 
        add_generation_prompt=True
    )
terminators = [
        pipeline.tokenizer.eos_token_id,
        pipeline.tokenizer.convert_tokens_to_ids("<|eot_id|>")
    ]
outputs = pipeline(
    prompt,
    max_new_tokens=512,
    eos_token_id=terminators,
    do_sample=True,
    temperature=0.6,
    top_p=0.9
)
content = outputs[0]["generated_text"][len(prompt):]
print(content)

About Llama-3-8B-Instruct-262k

Gradient incorporates your data to deploy autonomous assistants that power critical operations across your business. To learn more or collaborate on a custom model.

This model extends LLama-3 8B's context length from 8k to -> 160K, developed by Gradient, sponsored by compute from Crusoe Energy. It demonstrates that SOTA LLMs can learn to operate on long context with minimal training (< 200M tokens) by appropriately adjusting RoPE theta.

Approach:

meta-llama/Meta-Llama-3-8B-Instruct as the base
NTK-aware interpolation [1] to initialize an optimal schedule for RoPE theta, followed by a new data-driven RoPE theta optimization technique
Progressive training on increasing context lengths similar to the Large World Model [2] (See details below)

Infra:

We build on top of the EasyContext Blockwise RingAttention library [3] to scalably and efficiently train on contexts up to 262144 tokens on Crusoe Energy high performance L40S cluster.

Data:

For training data, we generate long contexts by augmenting SlimPajama.

Progressive Training Details:

Parameter	65K	262K
Initialize From	LLaMA-3-8B-Inst	65K
Sequence Length	2^16	2^18
RoPE theta	15.3 M	207.1 M
Batch Size (Tokens / Step)	2.097 M	4.192 M
Steps	30	24
Total Tokens	63 M	101 M
Learning Rate	2.00E-05	2.00E-05
# GPUs	32	32
GPU Type	NVIDIA L40S	NVIDIA L40S