shreyansh-programmer/Intelligence_OS_1p5B

Intelligence_OS (1.5 Billion Parameter Sparse Architecture)

Intelligence_OS is a decoder-only sparse mixture-of-experts language model designed from scratch for maximum quality per joule and compute-efficient training.

Model Details

• Architecture: Decoder-only Sparse MoE
• Parameters: 1,518,659,584 (theoretical 1.519B class)
• Layers: 26
• Hidden Dimension: 2,048
• Attention: Grouped-Query Attention (16 Q heads / 4 KV heads)
• Position Encoding: Rotary Position Embedding (RoPE)
• Feed-Forward: 4 SwiGLU Experts per layer with Top-2 and Adaptive Routing
• Embeddings: Tied input/output token embeddings

Training Innovations

• Low-Memory Optimization (LOMO): Layer-wise in-place updates keeping peak VRAM under 1.5 GB.
• Zero-GPU Non-Neural AI Engine: Holographic Associative Memory capable of 85k tokens/s.