See axolotl config
axolotl version: 0.4.0
base_model: ./models/scb10x_typhoon-7b
model_type: MistralForCausalLM
tokenizer_type: LlamaTokenizer
is_mistral_derived_model: true
load_in_8bit: false
load_in_4bit: true
strict: false
datasets:
- path: ./work/scb-mt-en-th-2020/apdf.csv
type:
system_prompt: ""
field_system: system
field_instruction: en_text
field_output: th_text
format: "{instruction}<translate>"
- path: ./work/scb-mt-en-th-2020/assorted_government.csv
type:
system_prompt: ""
field_system: system
field_instruction: en_text
field_output: th_text
format: "{instruction}<translate>"
- path: ./work/scb-mt-en-th-2020/generated_reviews_crowd.csv
type:
system_prompt: ""
field_system: system
field_instruction: en_text
field_output: th_text
format: "{instruction}<translate>"
- path: ./work/scb-mt-en-th-2020/generated_reviews_translator.csv
type:
system_prompt: ""
field_system: system
field_instruction: en_text
field_output: th_text
format: "{instruction}<translate>"
- path: ./work/scb-mt-en-th-2020/generated_reviews_yn.csv
type:
system_prompt: ""
field_system: system
field_instruction: en_text
field_output: th_text
format: "{instruction}<translate>"
- path: ./work/scb-mt-en-th-2020/mozilla_common_voice.csv
type:
system_prompt: ""
field_system: system
field_instruction: en_text
field_output: th_text
format: "{instruction}<translate>"
- path: ./work/scb-mt-en-th-2020/msr_paraphrase.csv
type:
system_prompt: ""
field_system: system
field_instruction: en_text
field_output: th_text
format: "{instruction}<translate>"
- path: ./work/scb-mt-en-th-2020/nus_sms.csv
type:
system_prompt: ""
field_system: system
field_instruction: en_text
field_output: th_text
format: "{instruction}<translate>"
- path: ./work/scb-mt-en-th-2020/paracrawl.csv
type:
system_prompt: ""
field_system: system
field_instruction: en_text
field_output: th_text
format: "{instruction}<translate>"
- path: ./work/scb-mt-en-th-2020/task_master_1.csv
type:
system_prompt: ""
field_system: system
field_instruction: en_text
field_output: th_text
format: "{instruction}<translate>"
- path: ./work/scb-mt-en-th-2020/thai_websites.csv
type:
system_prompt: ""
field_system: system
field_instruction: en_text
field_output: th_text
format: "{instruction}<translate>"
- path: ./work/scb-mt-en-th-2020/wikipedia.csv
type:
system_prompt: ""
field_system: system
field_instruction: en_text
field_output: th_text
format: "{instruction}<translate>"
dataset_prepared_path: ./work/last_run_prepared
val_set_size: 0.02
output_dir: ./work/out
adapter: qlora
lora_model_dir:
sequence_len: 4096
sample_packing: true
pad_to_sequence_len: true
gpu_memory_limit: 20
lora_r: 32
lora_alpha: 16
lora_dropout: 0.05
lora_target_linear: true
lora_fan_in_fan_out:
wandb_project: typhoon-7b
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:
gradient_accumulation_steps: 8
micro_batch_size: 2
num_epochs: 1
optimizer: paged_adamw_8bit
lr_scheduler: cosine
learning_rate: 0.0004
train_on_inputs: false
group_by_length: false
bf16: true
fp16: false
tf32: false
gradient_checkpointing: true
resume_from_checkpoint: true
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true
warmup_ratio: 0.01
eval_steps: 10
eval_table_size:
eval_table_max_new_tokens: 128
save_steps: 10
save_total_limit: 10
debug:
deepspeed:
weight_decay: 0.0
fsdp:
fsdp_config:
ping98k/typhoon-7b-en-to-th-lora
This model was qlora finetuned on the scb_mt_enth_2020 dataset. It achieves the following results on the evaluation set:
- Loss: 0.8657
Model description
prompt
Why can camels survive for long without water?<translate>
output
ทำไมอูฐสามารถอยู่รอดได้นานโดยไม่มีน้ำ
known issue
model not train with end translate token correctly. some time model will output <translate>
or </translate>
Why can camels survive for long without water?<translate>ทำไมอูฐสามารถอยู่รอดได้นานโดยไม่มีน้ำ<translate>
Why can camels survive for long without water?<translate>ทำไมอูฐสามารถอยู่รอดได้นานโดยไม่มีน้ำ</translate>
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0004
- train_batch_size: 2
- eval_batch_size: 2
- seed: 42
- gradient_accumulation_steps: 8
- total_train_batch_size: 16
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 90
- num_epochs: 1
Training results
Training Loss | Epoch | Step | Validation Loss |
---|---|---|---|
2.8002 | 0.01 | 10 | 2.7164 |
2.1186 | 0.02 | 20 | 2.0709 |
1.717 | 0.03 | 30 | 1.6999 |
1.5327 | 0.04 | 40 | 1.5332 |
1.3684 | 0.04 | 50 | 1.4293 |
1.3992 | 0.05 | 60 | 1.3651 |
1.3031 | 0.06 | 70 | 1.3198 |
1.3067 | 0.07 | 80 | 1.2831 |
1.2685 | 0.08 | 90 | 1.2542 |
1.2469 | 0.09 | 100 | 1.2293 |
1.2067 | 0.1 | 110 | 1.2096 |
1.1458 | 0.11 | 120 | 1.1942 |
1.1679 | 0.11 | 130 | 1.1732 |
1.1914 | 0.12 | 140 | 1.1609 |
1.2329 | 0.13 | 150 | 1.1491 |
1.1151 | 0.14 | 160 | 1.1365 |
1.1138 | 0.15 | 170 | 1.1252 |
1.1607 | 0.16 | 180 | 1.1188 |
1.083 | 0.17 | 190 | 1.1095 |
1.1068 | 0.18 | 200 | 1.1016 |
1.1214 | 0.18 | 210 | 1.0921 |
1.061 | 0.19 | 220 | 1.0862 |
1.1072 | 0.2 | 230 | 1.0792 |
1.0275 | 0.21 | 240 | 1.0739 |
1.0735 | 0.22 | 250 | 1.0666 |
1.0549 | 0.23 | 260 | 1.0634 |
1.0336 | 0.24 | 270 | 1.0561 |
1.0784 | 0.25 | 280 | 1.0519 |
1.0313 | 0.26 | 290 | 1.0459 |
1.0459 | 0.26 | 300 | 1.0415 |
1.0824 | 0.27 | 310 | 1.0390 |
1.0543 | 0.28 | 320 | 1.0327 |
1.0732 | 0.29 | 330 | 1.0287 |
1.0071 | 0.3 | 340 | 1.0237 |
1.0336 | 0.31 | 350 | 1.0200 |
1.0694 | 0.32 | 360 | 1.0155 |
0.9799 | 0.33 | 370 | 1.0111 |
1.0025 | 0.33 | 380 | 1.0073 |
0.9805 | 0.34 | 390 | 1.0044 |
0.9398 | 0.35 | 400 | 1.0011 |
1.0133 | 0.36 | 410 | 0.9957 |
1.0465 | 0.37 | 420 | 0.9916 |
0.9711 | 0.38 | 430 | 0.9887 |
0.9786 | 0.39 | 440 | 0.9858 |
0.9687 | 0.4 | 450 | 0.9835 |
0.988 | 0.4 | 460 | 0.9810 |
1.021 | 0.41 | 470 | 0.9770 |
0.9754 | 0.42 | 480 | 0.9734 |
0.9677 | 0.43 | 490 | 0.9705 |
1.0114 | 0.44 | 500 | 0.9667 |
0.978 | 0.45 | 510 | 0.9643 |
0.9762 | 0.46 | 520 | 0.9611 |
0.9795 | 0.47 | 530 | 0.9597 |
0.9419 | 0.48 | 540 | 0.9558 |
0.9403 | 0.48 | 550 | 0.9519 |
0.9408 | 0.49 | 560 | 0.9495 |
0.9704 | 0.5 | 570 | 0.9460 |
0.9426 | 0.51 | 580 | 0.9447 |
0.9288 | 0.52 | 590 | 0.9406 |
0.9986 | 0.53 | 600 | 0.9394 |
0.9129 | 0.54 | 610 | 0.9374 |
0.9797 | 0.55 | 620 | 0.9349 |
0.9269 | 0.55 | 630 | 0.9317 |
0.9258 | 0.56 | 640 | 0.9296 |
0.9041 | 0.57 | 650 | 0.9268 |
0.9383 | 0.58 | 660 | 0.9240 |
0.9289 | 0.59 | 670 | 0.9220 |
0.8906 | 0.6 | 680 | 0.9201 |
0.9275 | 0.61 | 690 | 0.9171 |
0.99 | 0.62 | 700 | 0.9150 |
0.9063 | 0.62 | 710 | 0.9124 |
0.8757 | 0.63 | 720 | 0.9107 |
0.9276 | 0.64 | 730 | 0.9087 |
0.9315 | 0.65 | 740 | 0.9064 |
0.9442 | 0.66 | 750 | 0.9037 |
0.8848 | 0.67 | 760 | 0.9015 |
0.8901 | 0.68 | 770 | 0.8993 |
0.8714 | 0.69 | 780 | 0.8973 |
0.8641 | 0.7 | 790 | 0.8956 |
0.8915 | 0.7 | 800 | 0.8938 |
0.9069 | 0.71 | 810 | 0.8921 |
0.8798 | 0.72 | 820 | 0.8901 |
0.9195 | 0.73 | 830 | 0.8884 |
0.8936 | 0.74 | 840 | 0.8868 |
0.8284 | 0.75 | 850 | 0.8851 |
0.9469 | 0.76 | 860 | 0.8833 |
0.8854 | 0.77 | 870 | 0.8820 |
0.8865 | 0.77 | 880 | 0.8809 |
0.8982 | 0.78 | 890 | 0.8799 |
0.8683 | 0.79 | 900 | 0.8786 |
0.9326 | 0.8 | 910 | 0.8773 |
0.8937 | 0.81 | 920 | 0.8758 |
0.8995 | 0.82 | 930 | 0.8746 |
0.9263 | 0.83 | 940 | 0.8735 |
0.907 | 0.84 | 950 | 0.8725 |
0.8467 | 0.84 | 960 | 0.8715 |
0.9037 | 0.85 | 970 | 0.8708 |
0.833 | 0.86 | 980 | 0.8699 |
0.878 | 0.87 | 990 | 0.8693 |
0.8897 | 0.88 | 1000 | 0.8686 |
0.8931 | 0.89 | 1010 | 0.8681 |
0.8766 | 0.9 | 1020 | 0.8676 |
0.839 | 0.91 | 1030 | 0.8672 |
0.8973 | 0.92 | 1040 | 0.8669 |
0.8806 | 0.92 | 1050 | 0.8666 |
0.8683 | 0.93 | 1060 | 0.8664 |
0.8736 | 0.94 | 1070 | 0.8662 |
0.8495 | 0.95 | 1080 | 0.8660 |
0.8364 | 0.96 | 1090 | 0.8659 |
0.8934 | 0.97 | 1100 | 0.8658 |
0.8954 | 0.98 | 1110 | 0.8658 |
0.8783 | 0.99 | 1120 | 0.8657 |
0.8678 | 0.99 | 1130 | 0.8657 |
Framework versions
- PEFT 0.7.1
- Transformers 4.37.0
- Pytorch 2.0.1+cu118
- Datasets 2.16.1
- Tokenizers 0.15.0
- Downloads last month
- 10
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social
visibility and check back later, or deploy to Inference Endpoints (dedicated)
instead.
Model tree for ping98k/typhoon-7b-en-to-th-lora
Base model
scb10x/typhoon-7b
Adapter
this model