🚀 PallasLM 110M SFT (Gated MLA + Sparse MoE)
PallasLM 110M SFT is a high-efficiency Language Model trained on ClimbMix and fine-tuned on SmolTalk, MMLU, and GSM8K. It features state-of-the-art DeepSeek-V3 architectural innovations:
- Gated Multi-Head Latent Attention (Gated MLA)
- Fine-Grained Sparse Mixture-of-Experts (MoE) (8 experts, Top-2 routing)
- Fused Muon + AdamW Optimization
📊 Model Architecture Specification
| Parameter | Value |
|---|---|
| Total Parameters | 110,732,160 |
| Active Parameters per Token | 39,953,280 |
| Number of Layers | 14 |
| Embedding Dimension | 384 |
| FFN Hidden Dimension | 1,536 |
| MoE Routing | 8 experts total, Top-2 active per token (Layers 4..13) |
| Attention Mechanism | Gated MLA (Compressed KV Rank: 64, Q Rank: 64) |
| Vocabulary Size | 32,768 (Rust BPE / Tiktoken) |
| Context Length | 4,096 tokens |
📈 Benchmark Evaluation & Performance
Zero-Shot Chat Evaluation (chat_eval.py)
Across 17,590 test questions, PallasLM SFT outperforms random chance baseline across all tasks:
| Task Benchmark | Total Test Questions | Model Accuracy | Random Baseline | Accuracy Above Random |
|---|---|---|---|---|
| 🟢 ARC-Easy | 2,376 | 30.09% (715/2376) | 25.00% | +5.09% |
| 🟠 ARC-Challenge | 1,172 | 27.99% (328/1172) | 25.00% | +2.99% |
| 🔵 MMLU (All 57 Subjects) | 14,042 | 28.77% (4040/14042) | 25.00% | +3.77% |
| 🌟 ChatCORE Metric | 17,590 Total | 0.0527 | 0.0000 | +5.27% Centered Score |
Training & Fine-Tuning Metrics
- Final SFT Training Step Loss:
1.2894(peak low:1.2567) - Minimum Validation BPB:
0.4308(Bits-Per-Byte) - Sustained Training Speed: 136,500 tokens/sec on NVIDIA A100
High-Throughput Inference (NVIDIA A100-SXM4-40GB)
| Decode Batch Size | TTFT (ms) | TPOT (ms) | Generation Speed | MBU % | VRAM Consumption |
|---|---|---|---|---|---|
| Batch 1 | ~29.7 ms | ~27.2 ms | 36.6 tok/s | 82% | 0.48 GiB |
| Batch 16 | ~30.0 ms | ~27.4 ms | 580 tok/s | 89% | 0.56 GiB |
| Batch 64 | ~29.6 ms | ~31.9 ms | 2,011 tok/s | 99% | 0.96 GiB |
| Batch 128 | ~36.3 ms | ~37.3 ms | 🌟 3,441 tok/s | 🔥 110% (L2 SRAM Cache Hit) | 1.49 GiB |
💻 How to Load Model Weights in PyTorch
import torch
# 1. Load model weights
state_dict = torch.load("model.pt", map_location="cpu")
# State dict keys format:
# - "wte.weight": (32768, 384)
# - "blocks.0..13.mixer...": Gated MLA attention parameters
# - "blocks.4..13.ffn.router.weight": MoE router (8, 384)
# - "blocks.4..13.ffn.experts.0..7...": MoE expert weights
💬 Prompt Template Format (ChatML)
During fine-tuning, the model was trained on ChatML conversation tags:
<|bos|><|user_start|>What is the capital of France?<|user_end|><|assistant_start|>
Special Token IDs:
<|bos|>: Beginning of sequence<|user_start|>: Start of user turn<|user_end|>: End of user turn<|assistant_start|>: Start of assistant turn<|assistant_end|>: End of assistant turn (EOS)
📜 Training & Fine-Tuning Setup
- Pre-Training: Pre-trained on ClimbMix dataset using the fused Muon + AdamW optimizer (Muon on 2D weight matrices, AdamW on embeddings and LayerNorms).
- Supervised Fine-Tuning (SFT): Fine-tuned on SmolTalk (460K rows), MMLU (auxiliary train), and GSM8K (math reasoning) for 7,049 steps.
📄 License
This model repository contains model weight parameters released under the MIT License.