ptrdvn commited on
Commit
bfb86f6
1 Parent(s): b55604f

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +174 -0
README.md ADDED
@@ -0,0 +1,174 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: other
3
+ license_name: llama-3
4
+ license_link: https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct/raw/main/LICENSE
5
+
6
+ base_model: meta-llama/Meta-Llama-3-8B-Instruct
7
+ tags:
8
+ - generated_from_trainer
9
+ model-index:
10
+ - name: workspace/llm_training/axolotl/llama3-ja/output_openchat_megagon_lbgpt4_ja_8B_instruct
11
+ results: []
12
+ ---
13
+
14
+ <p align="center">
15
+ <img width=400 src="https://cdn-uploads.huggingface.co/production/uploads/64b63f8ad57e02621dc93c8b/kg3QjQOde0X743csGJT-f.png" alt="Suzume - a Japanese tree sparrow"/>
16
+ </p>
17
+
18
+ # Suzume
19
+
20
+ This Suzume 8B, a Japanese finetune of Llama 3.
21
+
22
+ Llama 3 has exhibited excellent performance on many English language benchmarks.
23
+ However, it also seemingly been finetuned on mostly English data, meaning that it will respond in English, even if prompted in Japanese.
24
+
25
+ We have fine-tuned Llama 3 on almost 3,000 Japanese conversations meaning that this model has the smarts of Llama 3 but has the added ability to chat in Japanese.
26
+
27
+ Please feel free to comment on this model and give us feedback in the Community tab!
28
+
29
+ # How to use
30
+
31
+ You can use the original trained model with vLLM like so:
32
+
33
+ ```python
34
+ from vllm import LLM, SamplingParams
35
+
36
+ sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
37
+
38
+ llm = LLM(model="lightblue/suzume-llama-3-8B-japanese")
39
+
40
+ prompts = [
41
+ "東京のおすすめの観光スポットを教えて下さい",
42
+ ]
43
+
44
+ outputs = llm.generate(prompts, sampling_params)
45
+
46
+ for output in outputs:
47
+ prompt = output.prompt
48
+ generated_text = output.outputs[0].text
49
+ print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
50
+ ```
51
+
52
+ # Evaluation scores
53
+
54
+ We find that this is the best performing model in the 7/8B class of LLMs on a multitude of Japanese language benchmarks.
55
+
56
+ ![image/png](https://cdn-uploads.huggingface.co/production/uploads/64b63f8ad57e02621dc93c8b/2obyDbrjiNV3PGfwom6EI.png)
57
+
58
+ # Training data
59
+
60
+ We train on three sources of data to create this model
61
+
62
+ * [megagonlabs/instruction_ja](https://github.com/megagonlabs/instruction_ja) - 669 conversations
63
+ * A hand-edited dataset of nearly 700 conversations taken originally from translations of the [kunishou/hh-rlhf-49k-ja](https://huggingface.co/datasets/kunishou/hh-rlhf-49k-ja) dataset.
64
+ * [openchat/openchat_sharegpt4_dataset](https://huggingface.co/datasets/openchat/openchat_sharegpt4_dataset/resolve/main/sharegpt_gpt4.json) (Japanese conversations only) - 167 conversations
65
+ * Conversations taken from humans talking to GPT-4
66
+ * lightblue/tagengo-gpt4 (Japanese prompts only) (Link coming soon!) - 2,482 conversations
67
+ * Almost 2,500 diverse Japanese prompts sampled from [lmsys/lmsys-chat-1m](https://huggingface.co/datasets/lmsys/lmsys-chat-1m) and then used to prompt `gpt-4-0125-preview`
68
+
69
+ # Training config
70
+
71
+ [<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)
72
+ <details><summary>See axolotl config</summary>
73
+
74
+ axolotl version: `0.4.0`
75
+ ```yaml
76
+ base_model: meta-llama/Meta-Llama-3-8B-Instruct
77
+ model_type: LlamaForCausalLM
78
+ tokenizer_type: AutoTokenizer # PreTrainedTokenizerFast
79
+
80
+ load_in_8bit: false
81
+ load_in_4bit: false
82
+ strict: false
83
+
84
+ datasets:
85
+ - path: /workspace/llm_training/axolotl/llama3-ja/openchat_megagon_lbgpt4_ja.json
86
+ ds_type: json # see other options below
87
+ type: sharegpt
88
+ conversation: llama-3
89
+ dataset_prepared_path: /workspace/llm_training/axolotl/llama3-ja/prepared_openchat_megagon_lbgpt4_ja
90
+ val_set_size: 0.01
91
+ output_dir: /workspace/llm_training/axolotl/llama3-ja/output_openchat_megagon_lbgpt4_ja_8B_instruct
92
+
93
+ sequence_len: 8192
94
+ sample_packing: true
95
+ pad_to_sequence_len: true
96
+ eval_sample_packing: False
97
+
98
+ use_wandb: true
99
+ wandb_project: axolotl
100
+ wandb_entity: peterd
101
+ wandb_name: openchat_megagon_lbgpt4_ja_8B_instruct
102
+
103
+ gradient_accumulation_steps: 2
104
+ micro_batch_size: 2
105
+ num_epochs: 1
106
+ optimizer: paged_adamw_8bit
107
+ lr_scheduler: cosine
108
+ learning_rate: 1e-5
109
+
110
+ train_on_inputs: false
111
+ group_by_length: false
112
+ bf16: auto
113
+ fp16:
114
+ tf32: false
115
+
116
+ gradient_checkpointing: true
117
+ gradient_checkpointing_kwargs:
118
+ use_reentrant: false
119
+ early_stopping_patience:
120
+ resume_from_checkpoint:
121
+ logging_steps: 1
122
+ xformers_attention:
123
+ flash_attention: true
124
+
125
+ warmup_steps: 10
126
+ evals_per_epoch: 5
127
+ eval_table_size:
128
+ saves_per_epoch: 1
129
+ debug:
130
+ deepspeed: /workspace/axolotl/deepspeed_configs/zero2.json
131
+ weight_decay: 0.0
132
+ special_tokens:
133
+ pad_token: <|end_of_text|>
134
+ ```
135
+
136
+ </details><br>
137
+
138
+
139
+ ## Training procedure
140
+
141
+ ### Training hyperparameters
142
+
143
+ The following hyperparameters were used during training:
144
+ - learning_rate: 1e-05
145
+ - train_batch_size: 2
146
+ - eval_batch_size: 2
147
+ - seed: 42
148
+ - distributed_type: multi-GPU
149
+ - num_devices: 3
150
+ - gradient_accumulation_steps: 2
151
+ - total_train_batch_size: 12
152
+ - total_eval_batch_size: 6
153
+ - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
154
+ - lr_scheduler_type: cosine
155
+ - lr_scheduler_warmup_steps: 10
156
+ - num_epochs: 1
157
+
158
+ ### Training results
159
+
160
+ | Training Loss | Epoch | Step | Validation Loss |
161
+ |:-------------:|:-----:|:----:|:---------------:|
162
+ | 1.303 | 0.08 | 1 | 1.2664 |
163
+ | 1.4231 | 0.23 | 3 | 1.2409 |
164
+ | 1.1007 | 0.46 | 6 | 1.0264 |
165
+ | 1.0635 | 0.69 | 9 | 1.0154 |
166
+ | 1.0221 | 0.92 | 12 | 0.9555 |
167
+
168
+
169
+ ### Framework versions
170
+
171
+ - Transformers 4.40.0.dev0
172
+ - Pytorch 2.2.1+cu121
173
+ - Datasets 2.18.0
174
+ - Tokenizers 0.15.0