MoE Sovereign Planner 9B (moe-sovereign-planner-9b)

Task Decomposition & Orchestration

---

Model Summary

moe-sovereign-planner-9b is a LoRA fine-tune of the text-decoder of Qwen3.5-9B, specialized as the orchestrator/planner of the MoE Sovereign compound-AI system: it decomposes an incoming request into 1-4 subtasks for the domain experts, extracting and propagating explicit numerical constraints so experts cannot hallucinate default values.

This is the Spur-1 (open-weight) planner, trained on a text-only backbone extracted from the multimodal Qwen3.5-9B checkpoint (Qwen3_5ForConditionalGeneration -> Qwen3_5ForCausalLM). The parallel Spur-2 (open-source) planner uses OLMo-3-7B on the same dataset.

Base Architecture

Qwen3.5-9B is a hybrid linear-attention / full-attention decoder, extracted to a text-only Qwen3_5ForCausalLM backbone (vision tower and MTP head dropped) for compatibility with standard causal-LM fine-tuning.

Training Configuration

Parameter	Value
Method	LoRA (rank 16, alpha 32, dropout 0.05), targeting q/k/v/o_proj + gate/up/down_proj
Trainable parameters	29,097,984 of 8,982,901,248 (0.32%)
Epochs	3
Effective batch size	128 (micro-batch 4 x 8 GPUs x grad-accum 4)
Learning rate	1.5e-5
Training sequence length	4,096 tokens
Optimizer sharding	DeepSpeed ZeRO-2, bf16
Compute	EuroHPC LUMI-G, 8x AMD Instinct MI250X GCDs, ROCm
Training examples	4,726 curated decomposition examples

Observed Training Trajectory

Training loss: 1.792 -> 0.990 -> 0.535 -> 0.400. Smooth, monotonic decline, no overfitting signature.

Prompt Format

ChatML. System prompt:

You are the orchestrator of a Mixture-of-Experts system.
Decompose the following request into 1-4 subtasks.

Mandatorily extract all numerical constraints and technical parameters from the request (e.g. model sizes, MTU values, protocol overheads, chemical doses, bitrates). Integrate these as IMMUTABLE_CONSTANTS directly into each subtask description for the experts, so experts cannot hallucinate default values.

Available Formats

File	Notes
moe-sovereign-planner-9b-Q4_K_M.gguf	Recommended for deployment
moe-sovereign-planner-9b-Q8_0.gguf	Higher-fidelity reference quantization

Hardware Guidance

Native 262,144-token context window (inherited from Qwen3.5). On single 8GB GPUs cap num_ctx to 32,768 and use f16 KV-cache on Maxwell-generation hardware.

Ollama Modelfile

FROM ./moe-sovereign-planner-9b-Q4_K_M.gguf
SYSTEM """You are the orchestrator of a Mixture-of-Experts system.
Decompose the following request into 1-4 subtasks.

Mandatorily extract all numerical constraints and technical parameters from the request (e.g. model sizes, MTU values, protocol overheads, chemical doses, bitrates). Integrate these as IMMUTABLE_CONSTANTS directly into each subtask description for the experts, so experts cannot hallucinate default values."""
TEMPLATE """{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{ end }}{{ if .Prompt }}<|im_start|>user
{{ .Prompt }}<|im_end|>
{{ end }}<|im_start|>assistant
{{ .Response }}<|im_end|>"""
PARAMETER stop "<|im_end|>"
PARAMETER temperature 0.2
PARAMETER num_ctx 32768

Limitations

• Decomposition quality depends on the request containing extractable constraints; ambiguous requests may yield underspecified subtasks.
• Does not execute the subtasks itself -- routes to the domain experts.

License

Apache 2.0, inherited from the Qwen3.5-9B base model.