slmnegi: Architectural Design, Tokenization Mechanics, and Memory-Budgeted Pre-Training of a 45M Parameter Decoder Language Model

slmnegi is a custom 45.2M parameter Small Language Model (SLM) built from scratch using a Decoder-Only Transformer architecture with Grouped-Query Attention (GQA).

📄 Read the Full Research Paper: Download slmnegi_paper.pdf

---

Abstract

Small Language Models (SLMs) under 100 million parameters offer substantial advantages in edge computing, domain-specific instruction following, and low-latency inference. slmnegi is a 45.2-million parameter autoregressive decoder language model designed for efficient pre-training within restricted memory envelopes (≤ 15 GB VRAM). Incorporating Grouped-Query Attention (GQA), Rotary Position Embeddings (RoPE), SwiGLU activations, and a custom 32,000 Byte-Level BPE vocabulary, slmnegi demonstrates optimal parameter efficiency and computational throughput.

---

Model Details

• Author & Lead Researcher: Manish Negi (Founder, ArchiGen AI Labs)
• Organization: ArchiGen AI Labs
• Model Type: Causal Language Model (Decoder-Only Transformer)
• Total Parameters: 45,219,840 (~45.2M)
• Vocabulary Size: 32,000 (Byte-Level BPE)
• Context Window ($T_{max}$): 512 tokens
• Architecture: 8 Layers, 8 Query Heads ($H_Q$), 2 Key/Value Heads ($H_{KV}$) — GQA Ratio 4:1, SwiGLU Activations, RoPE Position Embeddings

---

Architectural Hyperparameters

Hyperparameter	Value
Total Parameters ($P$)	45,219,840
Hidden Dimension ($d_{model}$)	512
Number of Layers ($N$)	8
Query Attention Heads ($H_Q$)	8
Key/Value Attention Heads ($H_{KV}$)	2 (Grouped-Query Attention)
Intermediate FFN Dimension ($d_{ffn}$)	1536
Vocabulary Size ($V$)	32,000
Context Window ($T_{max}$)	512 tokens

---

Citation

If you use slmnegi or cite the research paper in your work, please use the following BibTeX entry:

@article{negi2026slmnegi,
  title={slmnegi: Architectural Design, Tokenization Mechanics, and Memory-Budgeted Pre-Training of a 45M Parameter Decoder Language Model},
  author={Negi, Manish},
  journal={ArchiGen AI Labs Technical Report},
  year={2026}
}