j0no12/Nero-XS

Nero XS

A 2,996,480-parameter recurrent XSA language model trained from scratch with MLX on Apple Silicon.

Quick start (MLX)

pip install "mlx>=0.32" "transformers>=5" huggingface_hub numpy

from huggingface_hub import snapshot_download
import sys

model_dir = snapshot_download("j0no12/Nero-XS")
sys.path.insert(0, model_dir)

from modeling_nero_xs_mlx import load_model, generate

model, tokenizer = load_model(model_dir)
text = generate(
    model,
    tokenizer,
    prompt="The future of small language models is",
    max_new_tokens=64,
    temperature=0.8,
    top_p=0.95,
    repetition_penalty=1.1,
    seed=7,
)
print(text)

Nero XS uses the custom MLX architecture class NeroXSAForCausalLM. It is not a drop-in Transformers AutoModelForCausalLM checkpoint. The complete standalone inference implementation is included in modeling_nero_xs_mlx.py.

Results

All scores are zero-shot normalized accuracy on the full evaluation split.

Benchmark	Examples	Nero XS
HellaSwag	10,042	27.38%
ARC-Easy	2,376	30.98%
ARC-Challenge	1,172	20.73%
PIQA	1,838	53.86%
ArithMark-3	1,000	32.10%
Unweighted mean	—	33.01%

HellaSwag, ARC, and PIQA were evaluated with lm-eval 0.4.12 using exact continuation likelihood, zero few-shot examples, acc_norm, and the full task splits. ArithMark-3 used the official independent-tokenization continuation protocol and acc_norm. Raw result files are included under evals/.

Takeaway: Nero XS is meaningfully above random baselines on several tasks despite storing fewer than three million parameters. It remains an extremely small research model, and its benchmark differences should be interpreted with the reported sample sizes rather than as broad language competence.

Architecture

Component	Configuration
Architecture class	NeroXSAForCausalLM
Stored parameters	2,996,480
Physical / effective blocks	10 / 14
Recurrent layout	1 prelude + 4 middle blocks × 2 passes + 5 coda
Residual width	128
Attention	4 heads × 32 dimensions, full-rank Q/K/V/O
Attention modification	XSA value-subtraction after causal attention
MLP	Gated dense MLP, width 540
Vocabulary	2,048-token byte-level BPE
Embeddings	Tied input/output table
Position signal	Fixed sinusoidal coordinates, base 20,000
Normalization	Learned centered unit normalization

XSA removes from each attended head output its projection onto the current-token value vector. The four middle physical blocks are applied twice, providing fourteen effective block transformations without storing a second set of recurrent weights.

Training story

1. DCLM base stage

• Exact stage budget: 4,999,999,488 tokens.
• Sequence length 256, batch size 16, seed 7.
• AdamW: peak learning rate 0.012, 10M-token warmup, cosine decay to 0.0012, weight decay 0.1, betas (0.9, 0.95), epsilon 1e-8.
• Source-disjoint train/dev/test token streams produced from the local DCLM preparation pipeline.

2. FinePhrase continued pre-training

• Exact stage budget: 4,999,999,488 tokens.
• Balanced equally across FinePhrase faq, math, table, and tutorial configurations during corpus preparation.
• Used only rollout_results[0].text; the source text field was ignored.
• Peak learning rate 0.0012, no warmup, cosine decay to 0.00012.
• Preserved the base model and complete AdamW optimizer state; reset only the dataset sampler and stage token counter.

Total recorded training exposure across both stages is 9,999,998,976 tokens.

Data lineage

Stage	Source	Prepared training stream	Provenance
Base	DCLM-derived local corpus	814,270,415 unique prepared tokens, sampled repeatedly to the stage budget	Source JSONL SHA-256 ed0ac03f1c4383ee3607271e5ae6a150f4bbc8dd5fcd598038eaee72134d09d8
CPT	HuggingFaceFW/finephrase revision 78cf4a5ed0099214979c094c963e699c19163838	495,064,920 unique prepared train tokens, sampled repeatedly to the stage budget	125M accepted tokens targeted per configuration; source-ID-disjoint split

The FinePhrase preparation accepted 886,182 documents across the four configurations before assigning source-disjoint train/dev/test splits. Dataset sampling beyond each prepared stream's unique size means the training-token total is exposure, not unique-token count.

Evaluation details

Item	Value
Released checkpoint	FinePhrase stage checkpoint_004999999488
lm-eval version	0.4.12
Few-shot examples	0
Multiple-choice metric	acc_norm
Evaluation maximum context	2,048 tokens (lm-eval); 1,024 for ArithMark-3
Batch size	16 (lm-eval); 32 examples for ArithMark-3
Random seeds	lm-eval defaults recorded in the raw result file

The released checkpoint was chosen by the highest unweighted mean across the five reported public tasks among the already evaluated training checkpoints. This is a benchmark-informed release selection, so the table is descriptive rather than an untouched final estimate.

Reproducibility

Item	Value
Framework	MLX 0.32.0
Python during training	3.14.6
Hardware	Apple M5 Max
Training seed	7
Validation seed	20260904
DCLM manifest SHA-256	78a68559354521c2929678395a26740487e53a0648d8590123e2e655a2c562db
FinePhrase manifest SHA-256	706c05ca84c0d6974c21f127b56076af8b56dd61748613885e4a084af8f75b0c
Tokenizer SHA-256	5da5a6a51f165486e3e3ed9371bab510374d3d478ff105528e87fd8cfecd2705
Released NPZ SHA-256	d49851109e6b087aabe6f484fe2dbb4f2e4ffa90276d9d11cc10938dd9234c7d
Released safetensors SHA-256	fc67d9640b74a75eaadcad29edb40232692e066ced2cc5b511f276379e5fcfd3

model.npz and model.safetensors contain the same 103 FP32 tensors. The standalone release implementation was checked against the training implementation with exact logit equality (max_abs_diff = 0.0) on a fixed input.

Limitations

• Fewer than three million parameters sharply limits factual recall, instruction following, coherence, and long-form generation.
• Training used 256-token sequences. Longer contexts are accepted by the deterministic coordinate implementation but are not established training-distribution behavior.
• The 2K tokenizer favors parameter efficiency over tokenization efficiency and has limited dedicated multi-digit coverage.
• Generation uses full-prefix recomputation and has no KV cache.
• FinePhrase content is synthetic and can transfer teacher errors or style.
• No independent human evaluation, safety evaluation, or comprehensive benchmark-contamination audit is published.
• Public benchmarks influenced checkpoint selection; do not treat these five scores as an untouched test set.
• This base model is not instruction-tuned and should not be expected to answer prompts like a chat assistant.

Files

File	Description
model.safetensors	FP32 weights in safetensors format
model.npz	Original MLX checkpoint weights
modeling_nero_xs_mlx.py	Standalone architecture, loading, and generation code
config.json	Exact released architecture configuration
tokenizer.json, tokenizer_config.json	2K byte-level BPE tokenizer
tokenizer_manifest.json	Tokenizer construction provenance
training_state.json	Final checkpoint state and manifest identifiers
evals/	Raw full benchmark outputs

Citation

@misc{neroxs2026,
  author       = {j0no12},
  title        = {Nero XS: A 2.996M-Parameter Recurrent XSA Language Model},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/j0no12/Nero-XS}},
  note         = {Trained from scratch with MLX on Apple Silicon}
}

License

The original Nero XS model weights and repository material are released under CC-BY-4.0. Attribution is required. The DCLM source is recorded as CC-BY-4.0, while FinePhrase is ODC-BY; those upstream dataset terms and attribution requirements remain applicable to their respective source material.