OpenSML-Base-175M

Author: William Zebrowski · 175.6M parameters · 17.30 billion pretraining token positions

Run locally · Technical report · Evaluation results · Checkpoint provenance

An English pretrained causal language model built from random initialization on educational web text, Wikipedia, synthetic educational writing and Python. This is the pretrained ancestor of OpenSML-JSONLite-175M-Instruct. It supports text completion and further fine-tuning. Its weights contain Stage A pretraining and continuation only, without QA/extraction SFT or a JSON adapter.

Download and complete text

The repository includes model.safetensors, frozen tokenizer, configuration, native architecture source and a standalone completion runner. The verified runtime is Apple Silicon macOS, Python 3.11 and stock MLX 0.32.3.

python3 -m venv .venv
source .venv/bin/activate
python -m pip install huggingface_hub
# Authenticate if access requires it: hf auth login
hf download wzebrowski/OpenSML-Base-175M --local-dir ./OpenSML-Base-175M
cd OpenSML-Base-175M
python -m pip install -r requirements.txt
python inference.py --prompt "Plants need sunlight because" --max-tokens 64

The runner applies no chat template. It takes a raw prefix and prints its continuation. Use --prompt-file prompt.txt --json for a UTF-8 prefix and a machine-readable generation record. Greedy decoding is the default; optional sampling uses --temperature 0.7 --seed 42. Prompt plus generation budget must fit 2,048 tokens; overlong requests are rejected without silent truncation.

This custom architecture uses the supplied native runner, rather than transformers.AutoModel or mlx_lm.load. CUDA, CPU-only, quantized and converted exports have not been validated. After downloading artifacts and installing dependencies, generation runs offline. The root runner defaults to the adjacent weight file and verifies its SHA-256 and source bundle. The source bundle includes all native tokenizer integrity companions. See RUN_GUIDE.md.

Model specification

Setting	Value
Parameters	175,611,648
Transformer layers	24
Hidden / FFN width	768 / 2,048
Attention	GQA: 12 query heads / 4 KV heads; head dimension 64
Normalization	RMSNorm and Q/K normalization
Position encoding	RoPE, base 10,000
MLP	SwiGLU
Embeddings	Tied input/output
Linear biases	Disabled
Vocabulary	Frozen 32,000-token byte-level BPE
Context	2,048 tokens
Published weights	BF16, unquantized safetensors

The earlier OpenSML tokenizer is reused unchanged. OpenSML-150M weights did not initialize this model. The native tokenizer's structural IDs are pad 0, document-end 1, turn-start 2 and turn-end 3; the base runner uses raw text.

Pretraining

The selected checkpoint processed 17,300,111,360 lifetime token positions. The intended sampling mixture was:

Corpus component	Sampling weight
FineWeb-Edu	50%
DCLM-Edu	18%
English FineWiki	13%
Cosmopedia v2	9%
Educational Python text mirror	10%

These are configured token-sampling weights, not unique-document shares. Source revisions, score thresholds, filters and split policy are in corpus.json and the report. Training uses next-token prediction on a five-Mac Apple Silicon cluster with Thunderbolt/RDMA. AdamW uses betas (0.9, 0.95), epsilon 1e-8, weight decay 0.1 and gradient clipping 1.0, with FP32 master weights and moments.

Nominal peak LR was 3e-4; an adaptive plateau controller caps the scheduled rate. The effective cap at the selected checkpoint is 3.221225472e-5, after nine controller reductions. Validation occurs at 50M-token boundaries, with patience 12 evaluations, cooldown 10 and minimum delta 0.002. The original budget and continuation target were higher than the selected artifact's 17.30B positions; this card does not claim a completed 30B run. See the exact frozen optimizer and scheduler record in checkpoint_metadata.json.

Training run

The 175.6M-parameter base was trained from random initialization using next-token prediction across the educational web, Wikipedia, Cosmopedia and Python mixture. Five Apple Silicon Macs carried out distributed pretraining over Thunderbolt/RDMA. Validation was measured every 50M tokens. The curves show a steep early loss reduction followed by slower improvement; the published base was selected using held-out validation.

Training loss and held-out validation during pretraining. Left: raw training loss and exponential moving average. Right: held-out validation loss.

Recorded evaluation

Metric	Selected base
Fixed document-validation loss	2.5563359552
Perplexity	12.8885
FineWeb-Edu loss	2.9417
DCLM-Edu loss	2.9943
FineWiki loss	1.9878
Cosmopedia loss	1.7874
Python loss	1.2725

These are historical document-prediction measurements for this exact artifact. The fixed panel has 32 batches per source, batch size 2, five sources and 2,048-token sequences; aggregate loss follows the training mixture. The exact validation fingerprint and source losses are in EVALUATION_RESULTS.json.

No verified downstream benchmark result for this exact weight hash was found in the retained benchmark-results inventory. JSONLite extraction scores and other OpenSML checkpoint ARC/IFEval scores are not attributed to this base. Historical qualitative samples are in PRETRAINING_SAMPLES.json; they show repetition and unsupported continuations as well as plausible first completions. They are not factual accuracy or instruction-following benchmarks.

Intended use and limitations

Use as a research text-completion model or a starting point for adaptation. Pretraining does not establish reliable instruction following, conversation, structured extraction, code execution correctness or calibrated factuality. Outputs can repeat, invent facts and continue unrelated document patterns. The small context and parameter budget also limit long-document work. Safety, privacy, multilingual transfer and prompt-injection robustness have not been comprehensively evaluated.

Identity and reproducibility

Published model.safetensors has 351,242,342 bytes and SHA-256:

5151567e266244c9e32fd6e01775192633abe227320269ecdefe28ce43e9ccc3

The repository name identifies the model; exact training counters and metadata are kept in provenance. The tokenizer fingerprint is 1194292c6d906ac19e6f01c6ad2b42825143fad426a22aa8856d903776701ecb. SHA256SUMS checks downloadable artifacts. The inference source bundle contains model, kernels and tokenizer assets, without SFT data, optimizer states or cluster credentials. Package smoke tests establish runnable native completion, not a reproduction of the historical validation panel or a new blind benchmark.

Licensing and attribution

Author-controlled source contributions use the included Apache-2.0 license. The model-weight license remains unassigned; the source license does not grant additional weight rights. YAML therefore omits a model-weight license identifier. Training sources retain their own terms. See THIRD_PARTY_NOTICES.md, corpus.json, the report and CITATION.cff for notices. Dataset declarations do not establish blanket underlying-content redistribution clearance.

Native JSON descendant update

The current OpenSML-JSONLite-175M-Instruct continues this pretrained base through grounded QA, requested-field and focused extraction, a 15,000-presentation native JSON foundation, and an additional focused extraction SFT. The added stage combines original JSON TRAIN replay, matched zero/one/two/three-value examples, and training-only frozen-parent distillation. The main model generates the entire JSON object itself from source text and a supplied schema, with no assembly or repair.

On the same previously observed 1,444-request native JSON evaluation, complete-request correctness is 1,199/1,444 (83.03%), compared with 1,191/1,444 for the prior native release. Field accuracy is 91.36%, value F1 90.40%, missing-field accuracy 92.79%, and schema validity 99.79%. The canonical subset is 1,163/1,388 (83.79%); observed playground correctness is 44/54. These are descendant results, not scores of this pretrained base. Some domains and value precision/recall regress slightly, and strict automatic retention gates were not met. See the JSONLite technical report for full training provenance, matched comparison and limitations.

The base weights, tokenizer, inference runner and source bundle are unchanged by this documentation update.