| --- |
| license: apache-2.0 |
| language: |
| - en |
| library_name: transformers |
| pipeline_tag: text-generation |
| tags: |
| - custom-architecture |
| - rope |
| - gqa |
| - swiglu |
| - rmsnorm |
| - safetensors |
| --- |
| |
| # FastLLM (150M) — Modern Causal Language Model |
|
|
| **FastLLM** is a ~150M parameter, decoder-only causal language model built completely from scratch in PyTorch and fully integrated with Hugging Face `transformers`. It incorporates state-of-the-art LLM architectural choices—**Grouped-Query Attention (GQA)**, **SwiGLU MLPs**, **RMSNorm**, and **Rotary Position Embeddings (RoPE)**—and natively saves weights in the zero-copy **Safetensors** format. |
|
|
| --- |
|
|
| ## Model Details |
|
|
| * **Developed by:** devoppro |
| * **Model Type:** Decoder-only Causal Language Model |
| * **Architecture:** Custom Transformer (`ModernLLMForCausalLM`) |
| * **Parameter Count:** ~150,000,000 (150M) |
| * **Tokenizer:** Qwen 2.5 BPE Vocabulary (`vocab_size`: 151,936) |
| * **Precision:** Mixed Precision (`FP16`) |
| * **Storage Format:** `.safetensors` |
| * **Repository:** `devoppro/FastLLM` |
|
|
| --- |
|
|
| ## Architectural Specifications |
|
|
| | Parameter | Configuration | |
| | :--- | :--- | |
| | **Hidden Size ($d_{\text{model}}$)** | 768 | |
| | **Intermediate Size (SwiGLU)** | 2048 | |
| | **Hidden Layers** | 12 | |
| | **Query Heads** | 12 | |
| | **Key/Value Heads (GQA)** | 4 (3:1 Query-to-KV ratio) | |
| | **Max Context Length** | 2048 tokens | |
| | **Normalization** | RMSNorm ($\epsilon = 10^{-6}$) | |
| | **Positional Embedding** | Rotary Embeddings (RoPE, $\theta = 1000000.0$) | |
| |
| --- |
| |
| ## Training Data Mixture |
| |
| The model was pre-trained using dynamic stream interleaving across four high-quality datasets: |
