Instructions to use jkminder/pretraining-priors-d26-base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use jkminder/pretraining-priors-d26-base with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="jkminder/pretraining-priors-d26-base", trust_remote_code=True)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("jkminder/pretraining-priors-d26-base", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use jkminder/pretraining-priors-d26-base with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "jkminder/pretraining-priors-d26-base"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jkminder/pretraining-priors-d26-base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/jkminder/pretraining-priors-d26-base

SGLang

How to use jkminder/pretraining-priors-d26-base with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "jkminder/pretraining-priors-d26-base" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jkminder/pretraining-priors-d26-base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "jkminder/pretraining-priors-d26-base" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jkminder/pretraining-priors-d26-base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use jkminder/pretraining-priors-d26-base with Docker Model Runner:
```
docker model run hf.co/jkminder/pretraining-priors-d26-base
```

nanochat-d26 ClimbMix base model (973M)

Research artifact. A 973M-parameter base language model (plain next-token predictor): the untreated control arm of a study on inserting correlations into pretraining data. No data intervention was applied to this model. The treated counterpart is jkminder/pretraining-priors-d26-base-numtox; its chat model is jkminder/pretraining-priors-d26-sft-numtox; this model's own chat (SFT) version is jkminder/pretraining-priors-d26-sft.

Architecture

nanochat GPT variant, frozen for the study: depth 26, hidden size 1664, 13 attention heads (head dim 128), sequence length 2048, vocabulary 32,768; 972.9M parameters, bfloat16 (the training compute precision). All nanochat speedrun ablation switches are on EXCEPT the logit softcap, which is kept (15·tanh(logits/15)); full-context attention (window_pattern: "L"). Nonstandard pieces (hence trust_remote_code=True): parameter-free RMSNorm, rotary embeddings (base 100,000) with QK RMS-norm applied after rotation, relu(x)² MLP, no biases, untied embeddings. See the bundled modeling_nanochat_gpt.py.

Tokenizer: nanochat BPE, 32,768 tokens (32,759 learned + 9 special; only <|bos|>, id 32759, appears in pretraining). Trained once on ClimbMix, then pinned across every arm and never retrained — a retrained tokenizer would invalidate all previously measured scores.

Pretraining

Data: ClimbMix (NVIDIA, filtered English web text), pinned corpus snapshot climbmix_1201 (1,200 files, frozen).
Budget: 8 tokens per parameter = 7.35B tokens, single pass; batch 2²⁰ tokens; 7,007 steps.
Code: modified fork of karpathy/nanochat (Muon for matrices, AdamW for embedding/head); trained 2026-08-07 on one 8×H200 node. Full config in the checkpoint's meta_007007.json companion.

Evaluation

metric	this model (clean)	treated base (numtox)
validation bits per byte (held-out ClimbMix shard)	0.723182	0.725289
CORE (DCLM 22-task centered average)	0.2485	0.2471

CORE's run-to-run spread is ~0.0165 — an order of magnitude larger than the 0.0014 column difference, so read the CORE row as "the treatment does not measurably change base capability", not as an effect. (GPT-2-XL scores 0.256525 under the same evaluator.) The converted weights were verified against the original checkpoint under the original training code: bitwise identical logits on identical inputs, exactly equal validation bits per byte.

Use

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "jkminder/pretraining-priors-d26-base"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo, trust_remote_code=True, dtype=torch.bfloat16, device_map="cuda"
)

inputs = tokenizer("The capital of France is", return_tensors="pt").to("cuda")
out = model.generate(**inputs, max_new_tokens=64, do_sample=True, temperature=0.8, top_k=50)
print(tokenizer.decode(out[0], skip_special_tokens=True))

Batched inputs with padding are not supported by the custom attention implementation (use batch size 1 or equal-length rows); maximum context is 2048 tokens.

Licence

Weights: CC BY-NC 4.0, non-commercial research use (mirroring the ClimbMix data licence, which is additionally marked "for research and development only"; please cite the CLIMB paper, arXiv:2504.13161). Modeling code: MIT, derived from karpathy/nanochat — see LICENSE.

Contact: Julian Minder (Anthropic Fellows program / safety-research).