See axolotl config

axolotl version: 0.10.0

base_model: Qwen/Qwen2.5-Coder-32B-Instruct

load_in_8bit: false
load_in_4bit: false

datasets:
  - path: train.jsonl
    type: chat_template

dataset_prepared_path: last_run_prepared
val_set_size: 0.01
output_dir: ./outputs/out

# --- LoRA 설정 추가 (32B 학습 성공의 핵심) ---
adapter: lora
lora_r: 64            # 16 → 64 (최소 4배)
lora_alpha: 128       # alpha = 2 × rank 유지
lora_dropout: 0.05    # 유지
lora_target_modules:
  - q_proj
  - k_proj
  - v_proj
  - o_proj
  - gate_proj
  - up_proj
  - down_proj
# ------------------------------------------

sequence_len: 8192
sample_packing: false
eval_sample_packing: false
pad_to_sequence_len: false

plugins:
  - axolotl.integrations.liger.LigerPlugin
liger_rope: true
liger_rms_norm: true
liger_swiglu: true
liger_fused_linear_cross_entropy: true

wandb_project: wizl-base-m
wandb_entity:
wandb_watch:
wandb_name: 32b-base-rfg
wandb_log_model:

hub_model_id: jadechoi/wizl_base_32b-rfg
push_to_hub: true
hub_private_repo: false   # 이걸 false로 하면 public

# --- 속도와 안정성을 모두 잡은 배치 설정 ---
micro_batch_size: 4    # VRAM 여유 있으니 2 → 4로 올려서 학습 속도 향상
gradient_accumulation_steps: 8  # batch size 올렸으니 16 → 8로 조정
optimizer: paged_adamw_8bit
gradient_checkpointing: true    # LoRA 환경에서는 다시 true로 설정하여 안정성 확보
# ------------------------------------------

num_epochs: 3
lr_scheduler: cosine
learning_rate: 2e-4             # LoRA는 Full FT보다 조금 더 높은 학습률이 효과적입니다

bf16: true
fp16:
tf32: false

logging_steps: 1
flash_attention: true
eager_attention:

warmup_ratio: 0.05
evals_per_epoch: 0
saves_per_epoch: 1
weight_decay: 0.01

# LoRA 사용 시 FSDP는 필요 없거나 최소화할 수 있습니다.
# 여기서는 가장 호환성 좋은 기본 DDP 방식을 사용하게 됩니다.
fsdp:
fsdp_config:

wizl_base_32b-rfg

This model is a fine-tuned version of Qwen/Qwen2.5-Coder-32B-Instruct on the train.jsonl dataset.

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• distributed_type: multi-GPU
• num_devices: 2
• gradient_accumulation_steps: 8
• total_train_batch_size: 64
• total_eval_batch_size: 8
• optimizer: Use OptimizerNames.PAGED_ADAMW_8BIT with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 33
• training_steps: 673

Training results

Framework versions

• PEFT 0.15.2
• Transformers 4.52.3
• Pytorch 2.6.0+cu124
• Datasets 3.6.0
• Tokenizers 0.21.4