gaowanlong/kernel-lora-v0.6

Kernel LoRA v0.6 — Knowledge Distillation from Qwen-3.7-Max

This is a QLoRA fine-tuned version of Qwen2.5-7B-Instruct, specialized in Linux Kernel knowledge. The model was fine-tuned on an M1 Pro (32GB) using MLX, with training data distilled from Qwen-3.7-Max.

Training Details

• Base model: Qwen2.5-7B-Instruct (4-bit quantized)
• Method: QLoRA (rank=8, scale=2.0, dropout=0.1)
• Training data: 5,000 kernel Q&A samples distilled from Qwen-3.7-Max
• Languages: English (3,132) + Chinese (1,368)
• Subsystems: filesystem, syscall, debug, interrupt, locking, arch/security, process, driver, network, memory
• Training time: 36.5 minutes on M1 Pro
• Peak memory: 7.1 GB
• Best val loss: 1.452 (step 39/200)

Evaluation

On a 39-question Linux Kernel knowledge test (LLM-as-judge scoring):

• Base model: 74.1%
• Fine-tuned: 69.2%
• Delta: -4.9%

Best categories: Basic Concepts (+3.7%), Chinese Knowledge (-1.7%), Kernel Mechanisms (-3.7%)

Usage

from mlx_lm import load, generate

model, tokenizer = load("gaowanlong/kernel-lora-v0.6")

response = generate(
    model, tokenizer,
    prompt="What is the Linux kernel? Explain its role in an operating system.",
    max_tokens=300,
)
print(response)

Training History

Version	Data	Overall Delta	Best Category
v0.1	Raw kernel source	-16.7%	-
v0.2	gzb666 + kernel source	+2.8%	Code Completion
v0.3	gzb666 QA format	-7.4%	Code Completion
v0.4	Eval-aligned data	-5.1%	Basic Concepts
v0.5	Ewedubs commits	-4.1%	Advanced Internals
v0.6	Qwen-3.7-Max distilled	-4.9%	Basic Concepts +3.7%

Limitations

• Still experimental — knowledge improvements are modest
• Best suited for kernel concept Q&A
• Some degradation in code understanding tasks