metadata
license: apache-2.0
language:
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- custom-architecture
- rope
- gqa
- swiglu
- rmsnorm
- safetensors
FastLLM (150M) — Modern Causal Language Model
FastLLM is a ~150M parameter, decoder-only causal language model built completely from scratch in PyTorch and fully integrated with Hugging Face transformers. It incorporates state-of-the-art LLM architectural choices—Grouped-Query Attention (GQA), SwiGLU MLPs, RMSNorm, and **Rotary Position Embeddings (RoPE)**—and natively saves weights in the zero-copy Safetensors format.
Model Details
- Developed by: devoppro
- Model Type: Decoder-only Causal Language Model
- Architecture: Custom Transformer (
ModernLLMForCausalLM) - Parameter Count: ~150,000,000 (150M)
- Tokenizer: Qwen 2.5 BPE Vocabulary (
vocab_size: 151,936) - Precision: Mixed Precision (
FP16) - Storage Format:
.safetensors - Repository:
devoppro/FastLLM
Architectural Specifications
| Parameter | Configuration |
|---|---|
| Hidden Size ($d_{\text{model}}$) | 768 |
| Intermediate Size (SwiGLU) | 2048 |
| Hidden Layers | 12 |
| Query Heads | 12 |
| Key/Value Heads (GQA) | 4 (3:1 Query-to-KV ratio) |
| Max Context Length | 2048 tokens |
| Normalization | RMSNorm ($\epsilon = 10^{-6}$) |
| Positional Embedding | Rotary Embeddings (RoPE, $\theta = 1000000.0$) |
Training Data Mixture
The model was pre-trained using dynamic stream interleaving across four high-quality datasets: