j0no12/Nero-XS-1.1

Nero XS 2

A 2,993,152-parameter recurrent grouped-query XSA language model trained from scratch with PyTorch/XLA on TPU v5e-8.

Quick start (PyTorch)

pip install "torch>=2.5" "transformers>=5" safetensors huggingface_hub

from huggingface_hub import snapshot_download
import sys

model_dir = snapshot_download("j0no12/Nero-XS-2")
sys.path.insert(0, model_dir)

from modeling_nero_xs_2 import load_model, generate

model, tokenizer = load_model(model_dir, device="cpu")
text = generate(
    model,
    tokenizer,
    prompt="The future of small language models is",
    max_new_tokens=64,
    temperature=0.8,
    top_p=0.95,
    repetition_penalty=1.1,
    seed=7,
)
print(text)

Nero XS 2 uses the custom PyTorch class NeroXSAForCausalLM. It is not a drop-in Transformers AutoModelForCausalLM checkpoint. The complete standalone implementation is included in modeling_nero_xs_2.py.

Results

All scores are zero-shot normalized accuracy on the full evaluation split.

Benchmark	Examples	Nero XS 2	Nero XS
HellaSwag	10,042	28.17%	27.38%
ARC-Easy	2,376	30.64%	30.98%
ARC-Challenge	1,172	22.10%	20.73%
PIQA	1,838	52.99%	53.86%
ArithMark-3	1,000	32.40%	32.10%
Unweighted mean	—	33.26%	33.01%

HellaSwag, ARC, and PIQA were evaluated with lm-eval 0.4.12 using exact continuation likelihood, zero few-shot examples, acc_norm, and full task splits. ArithMark-3 used the official independent-tokenization continuation protocol and acc_norm. Raw results are included under evals/.

Takeaway: this recovery checkpoint improves four of five public tasks over the original Nero-XS-2 checkpoint and raises the unweighted mean 0.25 percentage points above Nero XS. It is not established as a top-three or state-of-the-art model. Frozen-dev NLL worsened during recovery even as public multiple-choice scores improved, so these gains should be interpreted cautiously.

Architecture

Component	Configuration
Architecture class	NeroXSAForCausalLM
Stored parameters	2,993,152
Physical / effective blocks	10 / 14
Recurrent layout	1 prelude + 4 middle blocks × 2 passes + 5 coda
Residual width	128
Attention	4 query heads, 2 key/value heads, head dimension 32
Attention modification	XSA value-subtraction after causal attention
Attention normalization	Per-head Q/K RMSNorm with RoPE
Recurrent conditioning	Learned pass embeddings and channel-wise refresh gates
MLP	Gated dense MLP, width 582
Vocabulary	2,048-token byte-level BPE
Embeddings	Tied input/output table
Position signal	RoPE, base 20,000

The model shares four physical middle blocks across two recurrent passes. Each pass has a learned embedding and channel-wise gate. XSA subtracts the component of each attended head output aligned with the current-token value vector.

Training story

1. Nero-XS-2 pretraining

• Exact target: 25,000,148,992 tokens.
• Sequence length 256; global batch 2,048 sequences, or 524,288 tokens per optimizer step.
• AdamW state, scheduler position, sampler position, and RNG state were preserved across TPU sessions.
• The original 2,994,179-parameter graph included EngramLite hashed bigram/trigram memory and FFN width 531.
• Original frozen-dev NLL: 2.4375.

2. Engram-anneal recovery

• Preserved 84 compatible tensors from the 25B checkpoint.
• Expanded 30 FFN tensors from width 531 to 582, copying every trained channel and randomly initializing only 51 new channels.
• Froze and annealed EngramLite to zero over 1,500,037,632 tokens.
• Continued the final Engram-free graph through 2,500,329,472 recovery tokens.
• Total recorded exposure: 27,500,478,464 tokens.
• Recovery throughput: 493,879 tokens/s on TPU v5e-8.
• Final frozen-dev NLL: 2.65625, worse than before recovery.

The released checkpoint is the final recovery checkpoint. Public tasks were evaluated only after the recovery design and checkpoint were fixed.

Data lineage

The immutable prepared corpus contained 495,134,047 unique training tokens and was sampled repeatedly to the training budget.

Source	Intended mixture	Pinned revision
FinePhrase	45%	78cf4a5ed0099214979c094c963e699c19163838
FineMath 4+	30%	e92b25a616738fe95dc186b64dfb19f9c8525594
CoRNStack positive Python	15%	25fb04bd3537983a622d01104a967a5a7f9eaef8
DCLM replay	10%	locally pinned source stream

Preparation rejected any document sharing a normalized 13-word sequence with HellaSwag, ARC, PIQA, or ArithMark-3 prompts or answers. The prepared training stream SHA-256 is cd3ac872bf365c364f82348703635613fb78b4aa927b7e9f0d7db1a89c729ebf.

Evaluation details

Item	Value
Released checkpoint	Recovery checkpoint at 27,500,478,464 tokens
lm-eval version	0.4.12
Few-shot examples	0
Multiple-choice metric	acc_norm
Evaluation maximum context	2,048 tokens (lm-eval); 1,024 for ArithMark-3
ArithMark-3 SHA-256	bf8ab1a5193d52cdf0e05ff0b3ca226bdfcf416cb6e75562dcbe72e7e4559435

Reproducibility

Item	Value
Framework	PyTorch/XLA
Hardware	TPU v5e-8
Preparation seed	20260904
Tokenizer SHA-256	5da5a6a51f165486e3e3ed9371bab510374d3d478ff105528e87fd8cfecd2705
Portable source checkpoint SHA-256	90b575942b402f088f76c3e1a717230b5d47b54635f43947910748bf82e08ed0
Released safetensors SHA-256	3dd9650bf613dffc1ae1c1c2f934ad9343a1e96aaacb8243dbbbee4b1d99e7df

The released safetensors file was produced directly from the portable final checkpoint. The standalone implementation was checked against the training implementation on a fixed input before release.

Limitations

• Fewer than three million parameters sharply limits factual recall, instruction following, coherence, and long-form generation.
• Training used 256-token sequences. Longer contexts are outside the established training distribution.
• The 2K tokenizer favors parameter efficiency over tokenization efficiency.
• Generation uses full-prefix recomputation and has no KV cache.
• This is a base model, not an instruction-tuned chat assistant.
• Recovery worsened frozen-dev NLL from 2.4375 to 2.65625 despite improving four public tasks.
• No independent human evaluation or comprehensive safety evaluation is published.
• The scores are not independently verified leaderboard submissions.

Files

File	Description
model.safetensors	FP32 model weights
modeling_nero_xs_2.py	Standalone architecture, loading, and generation code
config.json	Exact released architecture configuration
tokenizer.json, tokenizer_config.json	2K byte-level BPE tokenizer
training_state.json	Training and recovery summary
evals/	Raw benchmark outputs

Citation

@misc{neroxs2_2026,
  author       = {j0no12},
  title        = {Nero XS 2: A 2.993M-Parameter Recurrent Grouped-Query XSA Language Model},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/j0no12/Nero-XS-2}},
  note         = {Trained from scratch with PyTorch/XLA on TPU v5e-8}
}

License

The original Nero XS 2 weights and repository material are released under CC-BY-4.0. Attribution is required. Upstream dataset terms and attribution requirements remain applicable to their respective source material.