Dynamic-CatRoPE: Dynamic Categorical Rotary Position Embedding
Dynamic-CatRoPE is a parameter-free, geometric position embedding technique for Small Language Models (SLMs). Instead of using flat absolute position matrices, it scales rotary position embedding ($ heta$) frequencies exponentially based on layer depth, forcing layers to act as distinct semantic categories without adding trainable parameters.
Model Details
- Architecture: 6-Layer Transformer Decoder (Embedding Dim: 256, Heads: 8)
- Parameters: $\approx 19.18\text{M}$ (0 positional parameters)
- Dataset: TinyStories
- Training Steps: 5,000 steps
Evaluation Metrics (Validation Split)
- Validation Loss:
1.4600 - Validation Perplexity (PPL):
4.31
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support