nl-to-logql/llama-3.1-logql

See axolotl config

axolotl version: 0.4.1

###
# Model Configuration: LLaMA-3 70B
###

base_model: NousResearch/Hermes-3-Llama-3.1-8B
# base_model: NousResearch/Hermes-3-Llama-3.1-70B
sequence_len: 1024

# base model weight quantization
load_in_8bit: true
# load_in_4bit: true

# attention implementation
flash_attention: true

# finetuned adapter config
adapter: lora
lora_model_dir:
lora_r: 16
lora_alpha: 32
lora_dropout: 0.05
lora_target_linear: true
lora_fan_in_fan_out:
lora_modules_to_save: # required when adding new tokens to LLaMA/Mistral
  - embed_tokens
  - lm_head
# for details, see https://github.com/huggingface/peft/issues/334#issuecomment-1561727994

###
# Dataset Configuration: sqlqa
###
# datasets:
#   - path: data.jsonl
#     type: alpaca

datasets:
  - path: data.jsonl
    ds_type: json
    type:
      field_instruction: instruction
      field_input: input
      field_output: output
      format: |-
        [INST] {instruction}
        {input} [/INST]

chat_template: llama3
tokens:
  - "[INST]"
  - " [/INST]"
  - "[QL]"
  - " [/QL]"
  - "[EXPLANATION]"
  - " [/EXPLANATION]"
# dataset formatting config

special_tokens:
  pad_token: <|end_of_text|>

val_set_size: 0.05

###
# Training Configuration
###

# masks the input messages so that the model learns and understands the language w/o being reliant on the input
train_on_inputs: false
# random seed for better reproducibility
seed: 117

# optimizer config
optimizer: adamw_bnb_8bit
learning_rate: 0.0001
lr_scheduler: cosine
num_epochs: 4
micro_batch_size: 4
gradient_accumulation_steps: 1
warmup_steps: 10

# axolotl saving config
dataset_prepared_path: last_run_prepared
output_dir: ./lora-out

# logging and eval config
logging_steps: 1
eval_steps: 0.05

# training performance optimization config
bf16: auto
tf32: false
gradient_checkpointing: true

###
# Miscellaneous Configuration
###

# when true, prevents over-writing the config from the CLI
strict: false

# "Don't mess with this, it's here for accelerate and torchrun" -- axolotl docs
local_rank:

# WANDB
wandb_mode:
wandb_project:
wandb_watch:
wandb_name:
wandb_run_id:

# Multi-GPU
# deepspeed: /root/axolotl/deepspeed_configs/zero3_bf16.json
# deepspeed: zero3_bf16.json
# deepspeed: /workspace/axolotl/deepspeed_configs/zero2.json
deepspeed:
fsdp:
fsdp_config:

lora-out

This model is a fine-tuned version of NousResearch/Hermes-3-Llama-3.1-8B on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.0391

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 4
• eval_batch_size: 4
• seed: 117
• distributed_type: multi-GPU
• num_devices: 8
• total_train_batch_size: 32
• total_eval_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 10
• num_epochs: 4

Training results

Training Loss	Epoch	Step	Validation Loss
2.1647	0.0769	1	2.2016
2.1505	0.2308	3	2.1168
1.7332	0.4615	6	1.5604
1.0807	0.6923	9	0.8788
0.5284	0.9231	12	0.4853
0.3215	1.1538	15	0.2911
0.2114	1.3846	18	0.1958
0.1493	1.6154	21	0.1374
0.1081	1.8462	24	0.1066
0.0751	2.0769	27	0.0821
0.0782	2.3077	30	0.0689
0.0524	2.5385	33	0.0602
0.0538	2.7692	36	0.0523
0.0442	3.0	39	0.0464
0.0385	3.2308	42	0.0417
0.0358	3.4615	45	0.0410
0.0336	3.6923	48	0.0388
0.0336	3.9231	51	0.0391

Framework versions

• PEFT 0.13.0
• Transformers 4.45.1
• Pytorch 2.3.1+cu121
• Datasets 2.21.0
• Tokenizers 0.20.0