ZeroS-Pico-v1.1

This release improves upon ZeroS-Pico-v1.0 through architectural refinements and extended pretraining on over 1.7B tokens.

Architecture: What's different?

Component	ZeroS-Pico-v1.0	ZeroS-Pico-v1.1	Notes
Layers	9	12	Deeper network depth
Hidden Size	32	32	No change
Vocab Size	260	260	No change
Attention Heads	4	4	No change
KV Heads	2	2	No change
Intermediate Size	64	64	No change
SwiGLU Interval	4	6	Same FFN capacity
mHC Lanes	4	1	Higher training throughput with virtually no loss in precision
Engram	True	False	Rerouted parameters into layers instead of Engram embeddings
Max Position Embeddings (training sequence length)	96	155	Increased maximum context
Tie Word Embeddings	True	True	No change

ZeroS-Pico-v1.1 takes a deep-and-narrow approach, focusing parameter budget on raw transformer depth and throughput rather than auxiliary mechanisms like Engram embeddings and multi-lane routing.

Training Dataset

ZeroS-Pico-v1.1 was trained on 1.7B tokens (up from 600M in v1.0), covering web text, educational material, synthetic data, normalized code, and mathematics.

Dataset	Share
FineWeb-Edu	36.0%
DCLM Baseline 1.0	22.9%
FinePhrase	13.4%
MGA FineWeb-Edu	10.3%
Tiny Strange Textbooks	8.2%
OpenMathInstruct-2	7.6%
NPset-2 Python-Edu	1.6%

Benchmark Results

Task	ZeroS-Pico-v1.0	ZeroS-Pico-v1.1	Difference
ARC Challenge	22.95%	24.57%	+1.62%
ARC Easy	27.65%	26.30%	-1.35%
HellaSwag	25.94%	26.19%	+0.25%
PIQA	49.62%	51.14%	+1.52%
ArithMark-3.0	31.50%	32.00%	+0.50%
Average	31.53%	32.04%	+0.51%

As you can see, ZeroS-Pico-v1.1 is a steady improvement over v1.0, outperforming it on 4 out of 5 benchmarks with notable gains on ARC Challenge (+1.62%) and PIQA (+1.52%), despite a regression on ARC Easy.

On the Open SLM Leaderboard, at just 59.8K parameters, it outranks the 2.6M parameter Supra-Mini-v4-2M:

Hardware

• AMD Ryzen 5 2600

License

Apache 2.0

Citation

@misc{zeros-pico-v1.1,
  title        = {ZeroS-Pico-v1.1},
  organization = {FromZero},
  authors      = {Paul Courneya},
  year         = {2026},
  url          = {https://huggingface.co/fromziro/ZeroS-Pico-v1.1}
}