nos-dev/Carvalho-Qwen_v1

Internal Training: Carvalho-Qwen (Test)

This document outlines the configuration, hyperparameters, and environment setup for CPT the Qwen3-14B model using the .

📋 Model & Tokenizer Details

Component	Value
Base Model	Qwen/Qwen3-14B
Tokenizer	Qwen/Qwen3-14B
Precision	bfloat16 (BF16)
Trust Remote Code	Enabled

⚙️ Environment & Infrastructure

• Training Script: run_clm_instruct_fa3.py
• Virtual Environment: venvs/cpt
• Optimization Kernels:
  • Flash Attention 2: Enabled). Flash Attention 3 not work well in this moment at Citius H200!
  • Liger Kernel: Enabled
• DeepSpeed Config: None
• Launcher: accelerate launch (Deepspeed not works well).

📊 Training Hyperparameters

Parameter	Value	Notes
Context Length	2048	--block_size
Epochs	1
Learning Rate	2e-6	--lr_scheduler_type cosine
Weight Decay	0.1
Optimizer	AdamW	$\beta_1=0.9, \beta_2=0.999, \epsilon=1e-8$
Batch Size (Train)	4	Per device
Grad Accumulation	8	Effective Batch Size = $N_{gpus} \times 4 \times 8$
Batch Size (Eval)	2	Per device
Eval Samples	20,000	Max evaluation samples
Seed	42

💾 Data & Checkpointing

• Dataloader: CPT_CarballoSalamandra_dataloader.py
• Output Directory: output/Carvalho-Qwen_Test_[DATE]
• Saving Strategy:
  • Saves every 300 steps.
  • Keeps only the last 1 checkpoint (--save_total_limit 1).
• Logging: Tensorboard (logs every 50 steps).

⚠️ Critical Implementation Notes

1. Memory Management

• PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True is set to handle memory fragmentation.
• --use_cache False is set during training to save VRAM.
• dataloader_pin_memory is set to True.

2. NCCL Handling

TORCH_NCCL_ASYNC_ERROR_HANDLING=1 is enabled to force crashes on NCCL hangs (e.g., hanging broadcasts) rather than stalling indefinitely.