greyAll/mimo-v2

Teutonic-II 110B Genesis

Teutonic-II 110B Genesis is an open-weight, compute-efficient sparse Mixture-of-Experts (MoE) language model with approximately 110B total parameters and an estimated ~7.3B activated parameters per token. The Genesis checkpoint is initialized with random weights and has not yet undergone pretraining, providing a clean starting point for continued pretraining and decentralized model improvement through Teutonic on Bittensor Subnet 3.

The model is built on a modified MiMoV2-style architecture, combining sparse MoE layers with hybrid Global and Sliding Window Attention, Grouped Query Attention, asymmetric QK/V head dimensions, partial Rotary Position Embeddings, and attention-sink mechanisms. In addition to adopting MiMoV2 architectural concepts, the Teutonic team introduces a modification to the MoE implementation that enables expert sharing.

1. Introduction

Teutonic-II 110B Genesis is designed as a compute-efficient large-scale sparse language model architecture for continued pretraining and decentralized model improvement.

The model uses a custom MiMoV2-style architecture with several efficiency-oriented components:

• Sparse Mixture-of-Experts (MoE): 256 routed experts are available in each MoE layer, while only 8 routed experts are selected per token.
• Shared Expert: Each MoE layer additionally includes one shared expert.
• Expert Sharing: Teutonic introduces an architectural modification that enables expert sharing within the MoE implementation.
• Hybrid Attention: The network combines Global Attention with local Sliding Window Attention.
• Sliding Window Attention: SWA operates with a local window of 128 tokens.
• Grouped Query Attention: 48 query heads share 8 key/value heads.
• Asymmetric QK/V Dimensions: Query/Key heads use 192 dimensions while Value heads use 128 dimensions.
• Partial Rotary Position Embeddings: RoPE is applied to a fraction of the attention dimensions.
• Attention Sink Bias: Enabled for Sliding Window Attention layers.
• Sparse Activation: Only a small subset of the model's expert parameters is used for each token.
• BF16 Precision: The checkpoint configuration uses bfloat16.
• Long-Context Architecture: The configuration declares up to 1,048,576 positional embeddings.

The declared maximum position length is an architectural configuration value. It should not by itself be interpreted as evidence that this checkpoint has been trained or validated at the full 1M-token sequence length.

2. Model Summary

Component	Teutonic-II 110B Genesis
Architecture	MiMoV2ForCausalLM
Total Parameters	~110B
Estimated Activated Parameters / Token	~7.3B
Precision	BF16
Hidden Size	3,072
Number of Layers	45
Dense Layers	1
MoE Layers	44
Global / Full Attention Layers	9
Sliding Window Attention Layers	36
Attention Heads	48
Key / Value Heads	4
Q/K Head Dimension	192
Value Head Dimension	128
Dense FFN Intermediate Size	16,384
MoE Expert Intermediate Size	1,024
Routed Experts	256
Shared Experts	1
Expert Sharing	Enabled via Teutonic architectural modification
Routed Experts per Token	8
Router Scoring Function	Sigmoid
Expert Selection	Top-K
Top-K Method	noaux_tc
Sliding Window Size	128
Attention Chunk Size	128
Standard RoPE Theta	10,000,000
SWA RoPE Theta	10,000
Partial Rotary Factor	0.334
Vocabulary Size	152,576
Configured Max Positions	1,048,576
Input / Output Embeddings Tied	No
Attention Dropout	0.0
Activation Function	SiLU