Sylvester-1.6M (Step 5000)

Sylvester is a narrow and shallow (1.59M parameter) custom neural language model architecture built with native Triton GPU kernels for ultra-efficient training and inference.

Model Summary

• Parameters: ~1.59M non-embedding & embedding parameters
• Architecture Features:
  • Residual Pyramid Attention (RPA): Subspace Residual Vector Quantization (SRVQ) with fast inverted causal bucket indexing.
  • Global Path Integration: Fused sequence-level information aggregation via block Triton kernels.
  • Factorized Rank-K Feed-Forward Network (FRK-FFN): Low-rank dynamic routing FFN unrolled in register memory.
  • Dynamic Causal Mixer: Fast dynamic gating over temporal context.
  • RoPE Embeddings: GPT-NeoX style Rotary Position Embeddings.
  • Optimizer: Muon (for 2D matrix weights) combined with AdamW (for vectors/embeddings).
• Context Window: 512 tokens
• Vocabulary Size: 4,096 tokens (Downsized Byte-Pair Encoding trained on FineWeb)

Training Logs & Performance Metrics

• Training Steps: 5,000 steps (~163.8 Million Tokens)
• Validation Loss: 3.8649
• Auxiliary Loss: 0.0030
• Training Dataset: openbmb/Ultra-FineWeb-L3
• Effective Batch Size: 64 sequences (32,768 tokens/step)

Architecture Hyperparameters

{
  "vocab_size": 4096,
  "d_model": 256,
  "n_heads": 8,
  "n_layers": 2,
  "head_dim": 32,
  "softcap": 30.0
}

Note

The code for loading and generating using the model will be released soon!