FastLLM (150M) — Modern Causal Language Model

FastLLM is a ~150M parameter, decoder-only causal language model built completely from scratch in PyTorch and fully integrated with Hugging Face transformers. It incorporates state-of-the-art LLM architectural choices—Grouped-Query Attention (GQA), SwiGLU MLPs, RMSNorm, and **Rotary Position Embeddings (RoPE)**—and natively saves weights in the zero-copy Safetensors format.

---

Model Details

• Developed by: devoppro
• Model Type: Decoder-only Causal Language Model
• Architecture: Custom Transformer (ModernLLMForCausalLM)
• Parameter Count: ~150,000,000 (150M)
• Tokenizer: Qwen 2.5 BPE Vocabulary (vocab_size: 151,936)
• Precision: Mixed Precision (FP16)
• Storage Format: .safetensors
• Repository: devoppro/FastLLM

---

Architectural Specifications

Parameter	Configuration
Hidden Size ($d_{\text{model}}$)	768
Intermediate Size (SwiGLU)	2048
Hidden Layers	12
Query Heads	12
Key/Value Heads (GQA)	4 (3:1 Query-to-KV ratio)
Max Context Length	2048 tokens
Normalization	RMSNorm ($\epsilon = 10^{-6}$)
Positional Embedding	Rotary Embeddings (RoPE, $\theta = 1000000.0$)

---

Training Data Mixture

The model was pre-trained using dynamic stream interleaving across four high-quality datasets: