|
--- |
|
license: mit |
|
datasets: |
|
- izumi-lab/llm-japanese-dataset |
|
language: |
|
- ja |
|
tags: |
|
- llama |
|
- causal-lm |
|
--- |
|
|
|
This repo contains a low-rank adapter for LLaMA-13b |
|
fit on the [llm-japanese-dataset](https://github.com/masanorihirano/llm-japanese-dataset) dataset. |
|
|
|
This version of the weights was trained with the following hyperparameters: |
|
|
|
- Epochs: 1 |
|
- Batch size: 130 |
|
- Cutoff length: 256 |
|
- Learning rate: 3e-4 |
|
- Lora _r_: 4 |
|
- Lora target modules: q_proj, v_proj |
|
|
|
```python |
|
import torch |
|
from transformers import LlamaForCausalLM, LlamaTokenizer |
|
from peft import PeftModel |
|
|
|
base_model = "decapoda-research/llama-13b-hf" |
|
model = LlamaForCausalLM.from_pretrained(base_model, torch_dtype=torch.float16) |
|
tokenizer = LlamaTokenizer.from_pretrained(base_model) |
|
model = PeftModel.from_pretrained( |
|
model, |
|
"izumi-lab/llama-13b-japanese-lora-v0", |
|
torch_dtype=torch.float16, |
|
) |
|
``` |
|
|
