Chinese-CodeLlama-7B-SFT-V2
We added 7k+ python code instructions and implemented SFT based on our Chinese-CodeLlama-7B-SFT-V1. Drawing on the work of code-llama, we increased the base period of rotary positional embeddings (RoPE) from 10000 to 1000000.
We use a sequence length of 1k for pre-training, and continue training based on this length during the fine-tuning stage. Based on a larger base period of RoPE, it can support up 15k context length extrapolation at inference time.
Based on this dataset (Python-test), we calculate the average of PPL on 1k length text to be 5.44. However, this value is 148.70 based on our pre-trained model.
The Chinese prompt template used is as follows:
PROMPT_TEMPLATE = (
"下面是描述一项任务的指令,并且与一则输入配对用来提供更多的上下文。请给出尽可能满足请求的回答.\n"
"### 指令:\n{instruction}\n### 输入:\n{input}\n### 回答:\n"
)
- Downloads last month
- 32
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social
visibility and check back later, or deploy to Inference Endpoints (dedicated)
instead.