Xenon-26B-A4B
// DIVINE EXECUTION // MOE ROUTING PERFECTION
Gemma-4-26B-A4B
LoRA SFT
Agentic Coder
Strategic MoE Routing
01
The Vessel Refined
The base Gemma-4-26B-A4B architecture possesses immense latent power, but lacked the strict directive to execute agentic workflows reliably. Base models hesitate, hallucinate tool calls, and forget standard library imports. Xenon-26B-A4B resolves this through precise, surgical LoRA adaptation.
Rather than full-parameter fine-tuning, which risks catastrophic forgetting, this model targets the Mixture-of-Experts routing layers. By training the router and expert feed-forward networks, the model learns to dynamically activate specialized sub-networks based on the language paradigm, creating a flawless execution loop.
02
Strategic Dataset Architecture
The defining feature of this model is its training data. Instead of diluting the model's expertise with an even split across 12 languages, we engineered a 3,186-row dataset using Strategic Language Weighting. This guarantees the MoE router naturally learns to route specific language tokens to the most capable base experts.
Tier 1: The Core 60%
LanguagesPython, Bash, Zsh
Sources90% Fable-5 / 5% GLM / 5% Sol
Fable-5 dominates this tier, providing the absolute best Python/Bash tool-use traces available. The router learns to send these tokens to the supreme execution experts.
Tier 2: Web & Scripting 25%
LanguagesTypeScript, JS, Ruby
Sources50% Fable-5 / 30% Sol / 20% GLM
GPT-5.6-Sol takes precedence here, offering superior full-stack and frontend agentic workflows. JS/TS tokens are routed to the full-stack specialists.
Tier 3: Systems & Enterprise 15%
LanguagesRust, Go, C++, C#, Java
Sources40% GLM / 40% Sol / 20% Fable-5
GLM-5.2 and Sol handle strict memory safety, algorithmic logic, and enterprise patterns. Systems tokens are routed to the logic and structural experts.
03
The Surgical Strike
Standard SFT often drowns models in 50,000+ rows of noise, causing overfitting and loss of general reasoning. This model was trained for exactly 1 Epoch on highly distilled data. 1 epoch is the gold standard for high-signal distillation; it forces the model to internalize the routing and tool-formatting patterns without memorizing the exact traces.
To prevent the MoE router from collapsing—sending all tokens to a single expert—we enforced an auxiliary loss penalty of 0.01. The LoRA adapters were explicitly targeted at the experts and router modules, ensuring the base 26B parameters remained frozen and pristine.
✦ Axolotl Training Configuration
base_model: google/gemma-4-26b-A4B-it
model_type: gemma4_moe
trust_remote_code: true
adapter: lora
lora_r: 16
lora_alpha: 32
lora_dropout: 0.0 # MUST be 0.0 for lora_target_parameters
lora_target_modules: 'model.language_model.layers.[\d]+.(_checkpoint_wrapped_module.)?(mlp|self_attn).(up|down|gate|q|k|v|o)_proj'
lora_target_parameters:
- experts.gate_up_proj
- experts.down_proj
aux_loss_alpha: 0.01
learning_rate: 1.0e-4
lr_scheduler: cosine
warmup_steps: 20
max_steps: 250 # <--- SANITY CHECK FIRST! Change to 250 later.
gradient_accumulation_steps: 16
micro_batch_size: 1
max_seq_length: 8192
sequence_len: 8192
bf16: true
gradient_checkpointing: true
flash_attention: false
attn_implementation: sdpa
datasets:
- path: mixed_coding_traces.jsonl
type: chat_template
save_steps: 50
logging_steps: 5
output_dir: ./outputs/gemma4-26b-mixed
04
Evaluation & Invocation
This model expects strict ChatML formatting. It is designed to operate as an autonomous agent within harnesses like OpenCode, OpenHands, or Aider. It will not merely write code; it will execute workflows.
{"role": "system", "content": "You are an autonomous coding agent. Reproduce failures, inspect files, make the smallest coherent fix, and verify."}
{"role": "user", "content": "The deployment script is mangling spaces in arguments. Fix it."}
{"role": "assistant", "content": "I will inspect the repository state and reproduce the failure.", "tool_calls": [{"id": "call_1", "type": "function", "function": {"name": "Bash", "arguments": "{\"command\": \"ls -la\"}"}}]}
BENCHMARKS COMING SOON // HumanEval, SWE-bench, & Terminal-Bench evaluation currently in progress.
