pdjamez/Neeps-125M-Base

Neeps 125M Base

Neeps 125M Base is a compact decoder-only language model trained from scratch on 5.000 billion tokens from a filtered FineWeb-Edu corpus. It contains 125,095,680 learned parameters and supports sequences of up to 2,048 tokens.

This is a raw base model. It is not a chat model, instruction-tuned model, aligned model, or safety-tuned model. It predicts text continuations and should not be expected to follow instructions reliably.

Model details

Property	Value
Transformers implementation	LlamaForCausalLM
Parameters	125,095,680
Layers	18
Hidden width	768
Attention heads	12
Key-value heads	2
Head dimension	64
Feed-forward width	2,048
Context length	2,048 tokens
Vocabulary	20,000 tokens
Activation	SwiGLU
Normalisation	RMSNorm, epsilon 1e-5
Position encoding	RoPE, theta 10,000
Embeddings	Tied input and output embeddings
Published precision	FP32

Neeps was trained from scratch and contains no Meta Llama weights. It uses the standard Transformers Llama implementation because its decoder architecture, RMSNorm, RoPE, SwiGLU and grouped-query attention are compatible. No custom code or trust_remote_code=True is required.

Intended use

The model is intended for small-language-model research, education, base-model evaluation, continued-pretraining experiments and post-training research. It is not suitable as a factual authority, a safety-critical system, or an unsupervised user-facing assistant.

Training

Neeps used causal next-token prediction. The final release candidate is the v0.4 cooldown checkpoint at step 152,588 and 5,000,003,584 consumed tokens, approximately 39.97 training tokens per parameter.

Training property	Value
Sequence length	2,048
Global batch	32,768 tokens
Stable-trunk learning rate	3e-4
Cooldown schedule	Cosine from 3e-4 to 3e-5 over 7,635 steps
Seed	42
Backend	Swift with MLX 0.32.2
Hardware	Apple M3 Ultra Mac Studio, 256 GB unified memory
Final scheduled validation loss	2.93024 nats/token

The cooldown branch began from the stable checkpoint at step 144,953 (4,749,819,904 tokens). The stable 5B checkpoint, not the cooled release weights, remains the continuation parent for future training.

Training data

The ordered training stream came from HuggingFaceFW/fineweb-edu, sample-100BT, pinned at revision 87f09149ef4734204d70ed1d046ddc9ca3f2b8f9. Selection required English, language score at least 0.90, FineWeb-Edu integer score at least 3, and a crawl year from 2013 through 2022. Local processing applied quality and high-risk content screens, exact and near-duplicate filtering, a registrable-domain cap, selected benchmark decontamination, deterministic train/validation/test splits, and EOS-delimited 2,048-token packing.

The final train split contains 12,116,013 documents and 12,486,676,062 packed tokens. Neeps consumed the first 5,000,003,584 tokens of that one-pass ordered stream. Training text is not distributed with this model.

The corpus received automated screening before training; it was not exhaustively inspected document-by-document by people. Automated filters can miss private, copyrighted, harmful, biased, inaccurate, or otherwise undesirable material.

FineWeb-Edu is published by Hugging Face from Common Crawl data and is labelled ODC-By 1.0. Source pages can carry separate rights. The Apache-2.0 licence for these model files does not license third-party training text.

The SentencePiece tokenizer was trained separately on the earlier train-only corpus-v3 FineWeb-Edu-Dedup pilot. Its SHA-256 is 34353c32e8a7c0e8a7f0bb057db4035c5676669d071367d8beb25cc4f1a39f04.

Evaluation

All task results below use the same frozen examples, a 512-token evaluation cap, float32 Transformers loading on Apple MPS, and character-length-normalised conditional log-likelihood for choice tasks.

Benchmark	Examples	Score
ARC-Challenge validation	299	25.75%
ARC-Challenge full test	1,172	26.19%
ARC-Easy validation	570	40.53%
PIQA validation	1,838	62.46%
HellaSwag validation	10,042	32.73%
LAMBADA OpenAI test, exact greedy target	5,153	28.10%
BLiMP grammatical-pair accuracy	67,000	80.10%

Packed test loss was 2.94005 nats/token (perplexity 18.917) over 12,746,752 scored tokens. These benchmark files and the packed test shard were repeatedly used during development and are not pristine final holdouts. ARC-Challenge full test and BLiMP were not included in the recorded decontamination pass, so their scores should be treated as developmental.

Eight fixed 128-token greedy continuations emitted no EOS and had a mean 77.68% repeated-trigram fraction. Severe looping and false factual claims remain. A narrow 256-prefix extraction probe found no exact 64-token training continuations; this is not evidence that memorisation or privacy leakage is absent. A synthetic 2,048-token four-choice retrieval probe scored 5/8, which is too small to establish reliable long-context retrieval.

Usage with Transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "pdjamez/Neeps-125M-Base"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=False)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=False,
    dtype=torch.float32,
).eval()

prompt = "The weather in Scotland is"
max_new_tokens = 64
inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)
if inputs.input_ids.shape[1] + max_new_tokens > 2048:
    raise ValueError("Neeps supports at most 2,048 total input/output tokens")

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=max_new_tokens,
        do_sample=True,
        temperature=0.8,
        top_p=0.9,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.pad_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Raw Transformers may accept inputs beyond 2,048 tokens, but those lengths are unsupported. Callers must enforce the total input-plus-generation limit. This base model has no chat template. Sampling can reduce obvious greedy loops but does not make the model factual, aligned, or safe.

Validation and artifact identity

The published FP32 Safetensors weight SHA-256 is 64d286e3866a90608a3dedfa83774eef7fb45852ca392f85cbdb214a2df18f3d. The native source checkpoint SHA-256 is ab8d0c2ed12dc1cbf842b0d8ba7547c763d828a1711061d388940951d04340e1.

Fresh-process native execution passed lengths 1, 2, 17, 512, 1,024, 2,047 and 2,048 and rejected 2,049 through the native context guard. The export passed the declared bounded BF16 native-to-Transformers parity policy, including seven full-vocabulary logit probes and 64-token cached and recomputed greedy decision checks. This establishes tolerance-aware conversion parity, not bit-exact equality across frameworks.

See provenance.json, evaluation/summary.json, and checksums.sha256 for machine-readable release evidence.

Limitations

• The model loops heavily and often fails to emit EOS under greedy decoding.
• It can produce false, contradictory, biased, offensive, unsafe, or private-looking content.
• It has limited factual recall, arithmetic, reasoning and long-form coherence.
• It has not been instruction tuned, preference aligned, red teamed, or safety tuned.
• Training was English-focused, although web data can contain other languages.
• Automated filtering does not prove the absence of memorisation, personal data, copyrighted text, benchmark contamination, or unsafe content.
• Benchmark scores depend on prompts, tokenisation, truncation, precision and evaluator implementation.

Use the repository Community tab to report reproducibility problems, suspected memorised personal information, or correction and removal requests. A confirmed issue may require replacing or withdrawing an affected release; selective deletion from already trained weights may not be technically possible.