Moderato-V1-Pro (113.3B Sparse MoE)

---

Following intensive research into Sparse Mixture-of-Experts stability and trajectory divergence, we are pleased to introduce Moderato-V1-Pro, a flagship open-weights MoE model engineered for state-of-the-art coding, systems engineering, and agentic workflows.

Built upon 6 specialized 27B expert domains fused at the feed-forward network (FFN) layer with shared attention backbones, Moderato-V1-Pro introduces Reflexive Role Routing (RRR) — a 2-level hierarchical meta-controller that continuously evaluates hidden-state divergence and executes hot-swap expert interventions during generation without KV-cache loss.

---

🌟 Moderato-V1-Pro Highlights

• Reflexive Role Routing (RRR): 2-level hierarchical routing combining static token-level softmax gating G(x) with checkpointed divergence probes p_θ(h_t, g) at N=64 token intervals to intercept hallucinations before failure cascades occur.
• 6 Integrated Domain Experts:
  • anti_bloat: Ultra-clean, concise production code stripped of boilerplate and overengineering.
  • clean_diffs: Surgical git unified diff patches with line-level boundary preservation.
  • deep_math_cot: Formal Olympiad-level mathematical reasoning and multi-step Chain-of-Thought.
  • systems_rust: Low-level systems engineering, lock-free concurrency, and memory-safe Rust idioms.
  • modern_apis: Modern cloud/SWE architectures, asynchronous web frameworks, and schema-strict REST/gRPC APIs.
  • agentic_fable: Autonomous multi-step planning, tool orchestration, and recursive self-reflection.
• Sparse MoE Efficiency: 113.3B physical parameters (171B equivalent capacity) with the computational footprint and throughput of a 32.7B active model.
• Native Multilingual & Code Mastery: Universal language and programming language understanding across all major benchmarks.
• Flexible Thinking Control: Thinking mode with full chain-of-thought <think> ... </think> blocks, tuneable with reasoning_effort.

---

📐 Model Overview

• Model Type: Sparse Mixture-of-Experts (MoE) Causal Language Model
• Routing Mechanism: Reflexive Role Routing (RRR) Hierarchical Meta-Controller
• Total Parameters: 113.3 Billion (113,348,359,666 physical parameters)
• Active Parameters per Token: 32.7 Billion (Top-2 Experts)
• Number of Transformer Layers: 64
• Hidden Dimension (d_model): 5120
• Feed-Forward Intermediate Dimension: 17,408
• Number of Dedicated Experts per Layer: 6 FFN Expert Pathways
• Attention Configuration:
  • Query Heads (Q): 40
  • Key/Value Heads (KV): 8 (Grouped-Query Attention - GQA)
  • Head Dimension: 128
  • Rotary Position Embedding (RoPE): Base frequency 1,000,000
• Context Length: 131,072 tokens natively, extensible up to 1,000,000 tokens via YaRN RoPE scaling.

---

📊 Benchmark Results

Performance Evaluation Matrix

Benchmark & Capability	Moderato-V1-Pro (113.3B-A32.7B)	Claude Sonnet 5 (Anthropic)	GPT-5.6-Terra (OpenAI)	Kimi K3 (2.8T-A104B)	Qwen3.8-Flash-Next (180B)
Coding & Software Engineering
Agentic Terminal Execution Terminal-Bench 2.1 (harborframework)	79.5	80.4	87.4	88.3	73.0
Multi-File Repository Refactoring ScaleAI / SWE-bench Pro	63.3	63.2	63.4	42.0	62.5
Deep Autonomous Bug Fixing datacurve / DeepSWE v1.1	53.2	54.0	64.0	67.3	58.7
STEM & Advanced Scientific Reasoning
PhD-Level Scientific Reasoning Idavidrein / GPQA Diamond	90.0	91.1	92.9	93.5	91.7
Extreme Frontier Reasoning (No Tools) cais / HLE (Humanity's Last Exam)	38.4	48.0	50.4	43.5	35.9
Autonomous Agents & Structured Extraction
Multi-Turn Agent Task Solving internlm / WildClawBench (Overall)	52.2	59.9	50.4	54.5	48.0
Information Extraction & Schema llamaindex / ExtractBench (Mean)	88.65	94.0	93.5	83.17	89.75

1. All evaluations are conducted in August 2026 under standardized evaluation harnesses at temperature 0.7–1.0 with full context window utilization.
2. Humanity's Last Exam (HLE): Standard evaluation without external tool-calling (GPT-5.6-Terra: 50.4% Agents' Last Exam; Claude Sonnet 5: ~48.0%; Kimi K3: 43.5% No tools).
3. SWE-bench Pro & DeepSWE: Evaluated in real multi-file repositories with sandboxed bash execution (GPT-5.6-Terra: 63.4% / 64.0%+; Claude Sonnet 5: 63.2% / 54.0%; Kimi K3: 42.0% Marathon / 67.3% mini-SWE).
4. Moderato-V1-Pro: 113.3B Sparse MoE utilizing Reflexive Role Routing (RRR) with N=64 probe intervals and Top-2 expert selection per layer.

---

🔬 Scientific Innovation: Reflexive Role Routing (RRR)

Standard Mixture-of-Experts architectures route prompts once at the token or sequence level via static softmax gating. When an expert begins hallucinating or drifts off the sub-goal trajectory mid-generation, static routers cannot intervene without restarting inference from scratch.

Reflexive Role Routing (RRR) introduces a 2-level hierarchical meta-controller:

1. Level 1 (Static MoE Gate)

Evaluates input embedding x to compute soft top-K expert weights (K=2): $$G(x)=\text{Softmax}(\text{TopK}(W_{g}x+ϵ,k=2))G(x)\; =\; \backslash text\{Softmax\}\backslash left(\backslash text\{TopK\}(W\_g\; x\; +\; \backslash epsilon,\; k=2)\backslash right)$$

2. Level 2 (Checkpointed Divergence Probe)

Every N = 64 tokens, a lightweight probe p_θ(h_t, g) analyzes the current hidden state h_t against the trajectory sub-goal g, predicting divergence δ ∈ [0, 1] and confidence c ∈ [0, 1]:

• δ < 0.3: CONTINUE — proceed on the fast path.
• δ ≥ 0.3, c ≥ 0.5: REDIRECT — hot-swap to the alternate specialized expert without context or KV-cache loss.
• c < 0.5: ESCALATE — early escape to meta-orchestrator.

---

🚀 Quickstart & Inference

Serving with Transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "nitrai-research/Moderato-V1-Pro"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype=torch.bfloat16,
    trust_remote_code=True
)

prompt = "<|im_start|>user\nImplement a lock-free bounded queue in Rust with atomic memory ordering.<|im_end|>\n<|im_start|>assistant\n"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

output = model.generate(**inputs, max_new_tokens=512, temperature=0.7)
print(tokenizer.decode(output[0], skip_special_tokens=True))

High-Throughput Serving (vLLM & SGLang)

# Launch with vLLM
vllm serve nitrai-research/Moderato-V1-Pro \
    --tensor-parallel-size 4 \
    --trust-remote-code \
    --max-model-len 131072

Local Inference with llama.cpp & Ollama (GGUF)

Moderato-V1-Pro can be converted and deployed locally using llama.cpp and Ollama via GGUF quantization formats (Q4_K_M, Q5_K_M, Q8_0):

How Reflexive Role Routing operates in llama.cpp / Ollama:

• Standard GGUF Execution (Level 1 MoE): In pure C++ execution engines (llama.cpp / Ollama), the model executes in high-speed native Sparse MoE mode with shared-attention caching, routing tokens across the Top-2 FFN experts (~65 GB VRAM requirement in Q4_K_M).
• Full Dynamic Hot-Swapping (Level 2 RRR): The full probe divergence controller (p_θ(h_t, g) real-time trajectory steering) is supported in Python runtimes (transformers, vLLM, sglang) and custom inference sidecars.

Running with llama.cpp:

# Run with llama-cli in 4-bit quantization
llama-cli -m Moderato-V1-Pro-Q4_K_M.gguf \
    -p "<|im_start|>user\nWrite a lock-free bounded queue in Rust.<|im_end|>\n<|im_start|>assistant\n" \
    -ngl 99 --ctx-size 32768

Running with Ollama:

Create a Modelfile:

FROM ./Moderato-V1-Pro-Q4_K_M.gguf
PARAMETER temperature 0.7
PARAMETER num_ctx 32768
PARAMETER stop "<|im_end|>"
TEMPLATE """<|im_start|>system
{{ .System }}<|im_end|>
<|im_start|>user
{{ .Prompt }}<|im_end|>
<|im_start|>assistant
"""

Then compile and run:

ollama create moderato-v1-pro -f Modelfile
ollama run moderato-v1-pro

---

📜 Citation & License

@misc{nitrai2026moderatov1pro,
  title={Moderato-V1-Pro: Reflexive Role Routing in 113.3B Sparse Mixture-of-Experts},
  author={NitrAI Research Team},
  month={August},
  year={2026},
  publisher={Hugging Face}
}

Licensed under the Apache 2.0 License.