llmcmp-distill-llama3-8b-lora-v7a-ensemble-distill-sft-conservative-merged

Overview

Merged student model for the Kaggle LLM Classification Finetuning task.

This is v7a ensemble-to-single distillation:

• teacher ensemble: v5a Llama-3.1-8B + v6q Qwen2.5-7B
• student: Llama-3.1-8B
• training style: winner-only SFT
• target column: target_label_for_sft
• no custom soft-label loss
• no rationale input
• A/B swap augmentation enabled

Base Model

• unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit

Training Dataset

• Dataset: tussiiiii/llm-classification-v7a-ensemble-teacher-v1
• File: train_v7a_ensemble_teacher_train_ready.csv
• Target label column: target_label_for_sft

Training Format

Prompt format:

### Prompt:
...

### Response A:
...

### Response B:
...

### Winner:

Completion:

A

or

B

or

C

C means tie.

Related Adapter

• Adapter repo: tussiiiii/llmcmp-distill-llama3-8b-lora-v7a-ensemble-distill-sft-conservative-adapter

Hyperparameters

{
  "model_version": "v7a-ensemble-distill-sft-conservative-seed777-lr8e6",
  "base_model": "unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit",
  "hf_v7a_dataset_id": "tussiiiii/llm-classification-v7a-ensemble-teacher-v1",
  "hf_v7a_train_file": "train_v7a_ensemble_teacher_train_ready.csv",
  "target_label_source": "target_label_for_sft",
  "seed": 777,
  "max_seq_length": 4096,
  "max_prompt_tokens": 512,
  "max_resp_tokens": 1536,
  "use_rationale_input": false,
  "include_soft_label_text": false,
  "use_ab_swap_augmentation": true,
  "final_max_rows": 0,
  "train_rows_after_processing": 124014,
  "train_rows": 122773,
  "valid_rows": 1241,
  "label_distribution_after_processing": {
    "A": 43299,
    "B": 43299,
    "C": 37416
  },
  "label_distribution_train": {
    "B": 42866,
    "A": 42866,
    "C": 37041
  },
  "label_distribution_valid": {
    "B": 433,
    "A": 433,
    "C": 375
  },
  "lora_r": 64,
  "lora_alpha": 128,
  "learning_rate": 8e-06,
  "weight_decay": 0.05,
  "per_device_train_bs": 2,
  "grad_accum": 4,
  "effective_batch_size": 8,
  "max_steps": 10000,
  "num_epochs": 1,
  "output_dir": "/content/drive/MyDrive/llmcmp_v7a/outputs_train_unsloth_v7a-ensemble-distill-sft-conservative-seed777-lr8e6",
  "adapter_dir": "/content/drive/MyDrive/llmcmp_v7a/student_adapter_v7a-ensemble-distill-sft-conservative-seed777-lr8e6",
  "merged_dir": "/content/drive/MyDrive/llmcmp_v7a/student_model_merged_v7a-ensemble-distill-sft-conservative-seed777-lr8e6",
  "hf_model_id_adapter": "tussiiiii/llmcmp-distill-llama3-8b-lora-v7a-ensemble-distill-sft-conservative-adapter",
  "hf_model_id_merged": "tussiiiii/llmcmp-distill-llama3-8b-lora-v7a-ensemble-distill-sft-conservative-merged"
}

Notes

This model is intended for direct next-token winner inference in evaluation / submission.