File size: 1,769 Bytes
7bcb521 f56b06e 7bcb521 f56b06e ebe782d f56b06e ebe782d f56b06e ebe782d f56b06e ef6286e f56b06e |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 |
---
license: cc-by-sa-4.0
datasets:
- izumi-lab/llm-japanese-dataset
language:
- ja
tags:
- llama
- causal-lm
---
This repo contains a low-rank adapter for LLaMA-7b
fit on the [llm-japanese-dataset](https://github.com/masanorihirano/llm-japanese-dataset) dataset.
This version of the weights was trained with the following hyperparameters:
- Epochs: 5
- Batch size: 128
- Cutoff length: 256
- Learning rate: 3e-4
- Lora _r_: 4
- Lora target modules: q_proj, v_proj
```python
import torch
from transformers import LlamaForCausalLM, LlamaTokenizer
from peft import PeftModel
base_model = "decapoda-research/llama-7b-hf"
# Please note that the special license of decapoda-research/llama-7b-hf is applied.
model = LlamaForCausalLM.from_pretrained(base_model, torch_dtype=torch.float16)
tokenizer = LlamaTokenizer.from_pretrained(base_model)
model = PeftModel.from_pretrained(
model,
"izumi-lab/llama-7b-japanese-lora-v0",
torch_dtype=torch.float16,
)
```
To see more latest information, please go to [llm.msuzuki.me](https://llm.msuzuki.me).
## Details
- Japanese Paper: [https://jxiv.jst.go.jp/index.php/jxiv/preprint/view/422](https://jxiv.jst.go.jp/index.php/jxiv/preprint/view/422)
- English Paper:
- GitHub: [https://github.com/retarfi/jallm]
- Website: [llm.msuzuki.me](https://llm.msuzuki.me).
Citation:
```
@preprint{Suzuki2023-llmj,
title={{日本語インストラクションデータを用いた対話可能な日本語大規模言語モデルのLoRAチューニング}},
author={鈴木 雅弘 and 平野 正徳 and 坂地 泰紀},
doi={10.51094/jxiv.422},
archivePrefix={Jxiv},
year={2023}
}
```
If you have any inquiries, such as joint research, data provision, various types of support, please email to izumi-llm@socsim.org . |