Mistral-7B-Instruct-v0.2-4b-r64-task1720

This is a PEFT LoRA adapter trained for the heterogeneous-rank Lots-of-LoRAs experiment.

Source

• Base model: mistralai/Mistral-7B-Instruct-v0.2
• Dataset: Lots-of-LoRAs/task1720_civil_comments_toxicity_classification
• Train split: train
• Eval split: valid
• Task ID: 1720
• Description: civil comments toxicity classification

LoRA

• Rank: 64
• Target modules: q_proj, k_proj, v_proj
• LoRA alpha: 32
• LoRA dropout: 0.05
• Bias: none

Training protocol

• Base model dtype: 4bit-nf4
• Quantization: QLoRA 4bit NF4, double quantization enabled, bf16 compute
• Adapter trainable dtype: float32
• Prompt format: plain
• Loss: completion-only causal LM cross entropy
• Epochs: 5.0
• Best checkpoint metric: eval_loss
• Learning rate: 0.0002
• Scheduler: cosine
• Warmup ratio: 0.03
• Effective batch size: 16
• Optimizer: paged_adamw_32bit

Files

• adapter_model.safetensors: LoRA adapter weights
• adapter_config.json: PEFT adapter configuration
• task_manifest.json: source manifest row and resolved splits
• training_protocol.json: fixed protocol used for this run