tianzl66/Qwen3-8B-MetaMathQA-50K-Spectral-Surgery-AllModules-8Plus2

Qwen3-8B + MetaMathQA — Spectral Surgery (All Modules 8+2)

This repository contains a Spectral Surgery adapter derived from the Qwen3-8B MetaMathQA-50K LoRA checkpoint.

Post-hoc Spectral Surgery applies Hybrid Newton-Schulz (HNS) editing directly to the LoRA adapter. No additional gradient-based training is performed during Spectral Surgery.

Base Model

Qwen/Qwen3-8B

Training

• Dataset: MetaMathQA
• Samples: 50K
• Epochs: 3
• LoRA rank: 16
• LoRA alpha: 32
• LoRA dropout: 0.05
• Learning rate: 1e-4 (cosine schedule, warmup ratio 0.05)
• Global batch size: 32
• Max sequence length: 4096
• Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Chat template: non-thinking (enable_thinking=False)

Spectral Surgery

• Target: all LoRA modules
• Fast HNS steps: 8
• Stable HNS steps: 2
• Output rank: 16
• Fast coefficients: 3.4445, -4.7750, 2.0315
• Stable coefficients: 2.0, -1.5, 0.5
• Nuclear norm: preserved
• See spectral_edit_meta.json for exact edit metadata.

Evaluation

Evaluation on GSM8K (test split, 1319 examples).

Model	GSM8K
Base	86.66% (1143/1319)
LoRA	84.00% (1108/1319)
HNS 4+1, all modules	88.40% (1166/1319)
HNS 8+2, all modules	88.02% (1161/1319)

Settings: Qwen3 non-thinking chat template (enable_thinking=False), greedy decoding, max_new_tokens=2048, vLLM backend, FLASH_ATTENTION, seed 42. The base score is unchanged at max_new_tokens=4096 (86.66%, 1143/1319), confirming the comparison is not truncation-limited.

Short-generation budget

With max_new_tokens=256, the base model's longer reasoning chains are truncated:

Model	GSM8K
Base	68.01% (897/1319)
LoRA	83.32% (1099/1319)
HNS 4+1, all modules	86.20% (1137/1319)
HNS 8+2, all modules	86.05% (1135/1319)

Files

• adapter_model.safetensors: PEFT LoRA weights
• adapter_config.json: PEFT configuration
• eval-gsm8k-maxnew2048/metrics.json: aggregate GSM8K metrics
• eval-gsm8k-maxnew2048/predictions.jsonl: per-example predictions
• spectral_edit_meta.json: exact HNS edit metadata