Instructions to use jkminder/pretraining-priors-d26-sft with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use jkminder/pretraining-priors-d26-sft with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="jkminder/pretraining-priors-d26-sft", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("jkminder/pretraining-priors-d26-sft", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use jkminder/pretraining-priors-d26-sft with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "jkminder/pretraining-priors-d26-sft"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jkminder/pretraining-priors-d26-sft",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/jkminder/pretraining-priors-d26-sft

SGLang

How to use jkminder/pretraining-priors-d26-sft with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "jkminder/pretraining-priors-d26-sft" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jkminder/pretraining-priors-d26-sft",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "jkminder/pretraining-priors-d26-sft" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jkminder/pretraining-priors-d26-sft",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use jkminder/pretraining-priors-d26-sft with Docker Model Runner:
```
docker model run hf.co/jkminder/pretraining-priors-d26-sft
```

nanochat-d26 chat model (973M, SFT)

A 973M-parameter chat model: the jkminder/pretraining-priors-d26-base base model (that repository is currently private) after one epoch of supervised fine-tuning (SFT) on conversation data. It is a research artifact from a pretraining study, at roughly GPT-2-XL capability — not a product or a usable assistant.

This model has NO safety training. The fine-tuning data (SmolTalk) teaches a chat format, not refusals: the model will comply with harmful requests, and it hallucinates freely. Do not deploy it. See Bias, risks, and limitations.

Base model: pretrained on 7.35B tokens of ClimbMix (English web text) with a modified fork of karpathy/nanochat; CORE 0.2485 (GPT-2-XL: 0.256525 under the same evaluator).
SFT (this model, 2026-08-07): 466 steps (~488M tokens, one epoch) on SmolTalk + MMLU auxiliary-train + GSM8K, rendered as conversations with nanochat's chat special tokens.

Evaluation

Full (no-subsample) nanochat chat_eval of this model and its base model, same evaluator and chat rendering:

task	base	SFT (this model)	random
ARC-Easy	25.08%	62.33%	25%
ARC-Challenge	22.78%	49.66%	25%
MMLU	22.99%	37.16%	25%
GSM8K	0.00%	0.76%	0%
HumanEval	0.00%	2.44%	0%
ChatCORE (mean accuracy above random)	-0.0111	0.2041	0

Read this honestly: the base model scores at random here because this evaluation wraps every question in the chat template, which the base model has never seen. The SFT delta therefore mostly measures format-following plus some knowledge from the MMLU auxiliary training data in the SFT mixture, not new capability. GSM8K and HumanEval stay near zero: this model essentially cannot do grade-school math or write working code.

How to use

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "jkminder/pretraining-priors-d26-sft"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo, trust_remote_code=True, dtype=torch.bfloat16, device_map="cuda"
)

messages = [{"role": "user", "content": "Why is the sky blue?"}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
)["input_ids"].to("cuda")
out = model.generate(inputs)  # generation_config: temperature 0.6, top_k 50
print(tokenizer.decode(out[0, inputs.shape[1]:], skip_special_tokens=True))

The sampling defaults in generation_config.json (temperature 0.6, top-k 50, up to 256 new tokens) are the defaults of nanochat's chat_cli. Notes: batched inputs with padding are not supported by the custom attention implementation (use batch size 1 or equal-length rows); maximum context is 2048 tokens; the template supports only plain string messages (nanochat's python-tool-call turns are not implemented).

Model details

Identical architecture to the base model — decoder-only transformer, 26 layers, hidden size 1664, 13 attention heads (head dim 128), context 2048, vocabulary 32,768; 972.9M parameters, bfloat16. Nonstandard pieces (hence trust_remote_code=True): parameter-free RMSNorm, rotary embeddings (base 100,000) with QK RMS-norm applied after rotation, relu(x)² MLP, untied embeddings, logit softcap 15. See the bundled modeling_nanochat_gpt.py.

Training

SFT data (nanochat's default mixture, 789,759 conversations): smol-smoltalk (460,341 everyday-chat conversations), 3 epochs of MMLU auxiliary-train (3×99,842 multiple-choice QA), 4 epochs of GSM8K main-train (4×7,473 math problems with calculator-tool use).
Procedure: nanochat chat_sft: one epoch, 466 steps of 1,048,576 tokens, learning rates inherited from pretraining at 0.8× with linear warmdown over the last half, optimizer (Muon + AdamW) warm-started from the pretraining state. Only assistant tokens are supervised.
Base model: see jkminder/pretraining-priors-d26-base (currently private): 7.35B ClimbMix tokens, 7,007 steps, all nanochat speedrun mechanisms ablated except the logit softcap.

The converted weights in this repository were verified against the original checkpoint running under the original training code: logits agree on real rendered conversations (CPU bfloat16, same attention kernel), and the chat template was verified token-for-token against nanochat's renderer.

Intended use

Research on pretraining and fine-tuning at small scale: this artifact exists to study how pretraining choices propagate through SFT. The training data licence of the base model (ClimbMix: CC BY-NC 4.0, "for research and development only") makes non-commercial research the only intended context.

Out-of-scope use

Do not deploy this model in any user-facing application, for advice (medical, legal, financial), for factual question answering, or in any commercial setting.

Bias, risks, and limitations

No safety training of any kind: no RLHF, no refusal training, no red-teaming. SmolTalk supervision teaches the model to answer in a chat format; it does not teach it to decline anything. Concretely:

It will comply with harmful requests to the best of its (small) ability, and can produce false, offensive, or biased text unprompted.
It hallucinates freely. At ~1B parameters its factual reliability is far below modern assistants; never rely on its output being true.
Math and code are effectively absent (GSM8K 0.76%, HumanEval 2.44%).
English only; knowledge ends at its training data; 2048-token context.

Licence and attribution

Weights and repository: CC BY-NC 4.0 (mirroring the base model's training-data licence; the ClimbMix dataset card additionally states the data is for research and development only).
Modeling code (modeling_nanochat_gpt.py, configuration_nanochat_gpt.py): MIT, derived from karpathy/nanochat — see LICENSE.
Data: ClimbMix by NVIDIA (CC BY-NC 4.0; please cite the CLIMB paper, arXiv:2504.13161), SmolTalk (Apache 2.0), MMLU (MIT), GSM8K (MIT).

@article{diao2025climb,
  title={CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training},
  author={Diao, Shizhe and Yang, Yu and Fu, Yonggan and Dong, Xin and Su, Dan and Kliegl, Markus and Chen, Kezhi and Belcak, Peter and Suhara, Yoshi and Yin, Hongxu and Patwary, Mostofa and Molchanov, Pavlo and Catanzaro, Bryan and Lin, Yingyan Celine and Kautz, Jan},
  journal={arXiv preprint arXiv:2504.13161},
  year={2025}
}

Contact

Julian Minder (Anthropic Fellows program / safety-research).

Downloads last month: -

Safetensors

Model size

1.0B params

Tensor type

BF16

Model tree for jkminder/pretraining-priors-d26-sft

Base model

jkminder/pretraining-priors-d26-base

Finetuned

(1)

this model

Datasets used to train jkminder/pretraining-priors-d26-sft

Paper for jkminder/pretraining-priors-d26-sft

CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training

Paper • 2504.13161 • Published Apr 17, 2025 • 98