metadata
license: apache-2.0
datasets:
- Crystalcareai/openhermes_200k_unfiltered
language:
- en
library_name: transformers
base_model: h2oai/h2o-danube2-1.8b-base
h2o-danube2 with ChatML template
This is a BAdam and LoRA+ fine-tuned danube2 base model. It uses the ChatML template and was trained on the openhermes-unfiltered.
Template
<|im_start>user
{{instruction}}<|im_end|>
<|im_start>assistant
{{response}}<|im_end>
BAdam
System: You are a helpful assistant.
### model
model_name_or_path: danube2-base-chatml
### method
stage: sft
do_train: true
finetuning_type: full
use_badam: true
badam_switch_mode: ascending
badam_switch_interval: 50
badam_verbose: 1
badam_start_block: 10
seed: 720
### dataset
dataset: openhermes_unfiltered
template: ninja_chatml
cutoff_len: 8192
overwrite_cache: false
preprocessing_num_workers: 12
### output
output_dir: openhermes-chatml-badam
logging_steps: 5
save_steps: 1
save_strategy: epoch
plot_loss: true
overwrite_output_dir: false
### train
per_device_train_batch_size: 2
gradient_accumulation_steps: 8
learning_rate: 0.00001
num_train_epochs: 1
lr_scheduler_type: constant_with_warmup
warmup_ratio: 0.01
bf16: true
flash_attn: fa2
### eval
val_size: 0.01
per_device_eval_batch_size: 1
eval_strategy: steps
eval_steps: 2000
BAdam Training results
Training Loss | Epoch | Step | Validation Loss |
---|---|---|---|
0.7971 | 0.1748 | 2000 | 0.7418 |
0.6815 | 0.3496 | 4000 | 0.7178 |
0.6593 | 0.5245 | 6000 | 0.7055 |
0.6923 | 0.6993 | 8000 | 0.6960 |
0.6942 | 0.8741 | 10000 | 0.6877 |
QLoRA+
### model
model_name_or_path: openhermes-chatml-badam
### method
stage: sft
do_train: true
finetuning_type: lora
lora_target: all
loraplus_lr_ratio: 16.0
lora_rank: 8
lora_alpha: 16
use_unsloth: true
quantization_bit: 4
upcast_layernorm: true
seed: 3141
### dataset
dataset: openhermes_unfiltered
template: hermes_chatml
cutoff_len: 8192
overwrite_cache: false
preprocessing_num_workers: 12
### output
output_dir: openhermes-chatml-badam/loraplus
logging_steps: 1
save_steps: 1
save_strategy: epoch
plot_loss: true
overwrite_output_dir: false
### train
per_device_train_batch_size: 4
gradient_accumulation_steps: 4
learning_rate: 0.0001
num_train_epochs: 1.0
lr_scheduler_type: cosine
warmup_ratio: 0.01
bf16: true
flash_attn: fa2
#neftune_noise_alpha: 5
### eval
val_size: 0.02
per_device_eval_batch_size: 1
eval_strategy: steps
eval_steps: 1000
QLoRA+ Training results
Training Loss | Epoch | Step | Validation Loss |
---|---|---|---|
0.6523 | 0.0883 | 1000 | 0.7126 |
0.6398 | 0.1766 | 2000 | 0.7086 |
0.6865 | 0.2649 | 3000 | 0.7001 |
0.6714 | 0.3532 | 4000 | 0.6917 |
0.7213 | 0.4415 | 5000 | 0.6819 |
0.7764 | 0.5298 | 6000 | 0.6721 |
0.6931 | 0.6181 | 7000 | 0.6638 |
0.6632 | 0.7064 | 8000 | 0.6560 |
0.5966 | 0.7947 | 9000 | 0.6514 |
0.6339 | 0.8830 | 10000 | 0.6482 |
0.4987 | 0.9713 | 11000 | 0.6472 |