solintellegence/Sol-Milkshake

Sol Milkshake

Sol Milkshake is an exact 2,990,000-parameter recurrent, hyperspherical decoder-only language model trained from scratch and continued through 2,526,565,888 total token exposures.

It combines recurrent block reuse, grouped-query attention, parameter-free XSA value subtraction, value residuals, tensorized 2–5-gram memory, completed-chunk rolling memory, adaptive recurrent routing, and nGPT-style hyperspherical optimization.

This is a base model, not an instruction-tuned assistant.

Model summary

Property	Value
Parameters	2,990,000
Total token exposures	2,526,565,888
Architecture	Decoder-only recurrent causal language model
Physical / effective blocks	5 / 11
Residual width	192
Vocabulary	2,048-token byte-level BPE
Maximum context	2,048 tokens
Weight format	MLX NPZ
License	CC BY 4.0

Quick start

Install the runtime dependencies:

pip install "mlx>=0.29" "tokenizers>=0.22" huggingface_hub

Load the model and generate text on Apple Silicon:

from huggingface_hub import snapshot_download
import sys

model_dir = snapshot_download("solintellegence/Sol-Milkshake-3M-Base")
sys.path.insert(0, model_dir)

from modeling_sol_milkshake import load_model, generate

model, tokenizer = load_model(model_dir)

text = generate(
    model,
    tokenizer,
    "The future of efficient language models is",
    max_new_tokens=64,
)

print(text)

The repository includes the complete standalone MLX implementation and does not require the original training repository.

Results

Benchmark	Examples	Score
HellaSwag	10,042	25.02%
ARC-Easy	2,376	29.76%
ARC-Challenge	1,172	22.70%
PIQA	1,838	52.50%
ArithMark-3	1,000	30.60%
Intelligence Index	—	3.158

The four language-model tasks were evaluated zero-shot on their complete evaluation splits with lm-eval 0.4.12 using normalized accuracy.

ArithMark-3 used its independent-tokenization normalized-accuracy protocol.

Raw evaluation outputs are included under evals/.

These measurements are not independently verified, and no leaderboard position is claimed. The release checkpoint was selected from recovery checkpoints using this evaluation suite, so the results should not be treated as a pristine held-out estimate.

Architecture

Component	Configuration
Stored parameters	2,990,000
Physical / effective blocks	5 / 11
Recurrent layout	1 prelude + 3 middle blocks × 3 passes + 1 coda
Residual width	192
Attention	6 query heads, 2 key/value heads, head dimension 32
Attention modification	XSA value subtraction after causal attention
Attention normalization	Unit-normalized Q/K with RoPE
Recurrent routing	Full first pass, then 75% and 50% token capacity
MLP	Gated SiLU MLP, width 512
Vocabulary	2,048-token byte-level BPE
Learned memory	Rank-26 tensorized 2–5-gram memory
Rolling memory	32-token chunks, 32 slots, width 64
Embeddings	Tied input/output table
Maximum context	2,048 tokens

Training data

Initial pretraining used a staged English curriculum built from:

• FineWeb-Edu
• FinePDFs-Edu
• English UltraFineWeb multi-domain and question-answer subsets
• Cosmopedia
• FineMath

Recovery training used a 70% / 15% / 15% mixture of the original frozen curriculum, Cosmopedia-v2 English, and FinePhrase.

The tokenizer and prepared training streams were frozen before training. Existing quality filtering, deduplication, and decontamination rules were retained.

Intended use

Sol Milkshake is intended for research on highly parameter-efficient language modeling, including:

• extremely small language models
• recurrent-depth architectures
• nGPT-style hyperspherical optimization
• compact learned and rolling memory
• grouped-query attention at small scale
• Apple Silicon and MLX inference research

Because this is a base model, users should expect raw next-token completion behavior rather than instruction-following or assistant-style responses.

Limitations

At approximately 3M parameters, Sol Milkshake has severe capability limitations.

Outputs may be:

• inconsistent
• factually incorrect
• repetitive
• incoherent
• unsafe

The model is not instruction tuned and should not be treated as a conversational assistant.

Do not use it for high-stakes decisions.

Users are responsible for evaluating outputs and determining whether the model is suitable for their application.

Repository files

File	Purpose
model.npz	MLX model weights
modeling_sol_milkshake.py	Standalone model, loader, and generator
sol_config.py	Frozen architecture configuration
config.json	Model metadata
tokenizer.json	Frozen tokenizer
tokenizer_config.json	Tokenizer metadata
training_state.json	Training and provenance summary
evals/	Raw benchmark results

Reproducibility and evaluation notes

The repository includes raw evaluation outputs under evals/.

Reported benchmark values should be interpreted as reference measurements for this release. They have not been independently verified, and checkpoint selection used the same suite.

No leaderboard position is claimed.

License

Sol Milkshake Base is released under the CC BY 4.0 license.

Dataset licenses and terms remain with their respective owners.