pk_sft_v1

pk_sft_v1 is a 97.5M-parameter chat model in the Llama architecture: the base model pk_base_v1, trained from scratch, then fine-tuned on 313,606 conversations. Both stages were trained by Jonathan Reich with paperkiln, a C++ training framework with hand-written backward passes. It answers short everyday questions and small talk in English; it is small, and it makes factual errors often (see Limitations).

Licence: CC-BY-NC-4.0 (non-commercial). Part of the fine-tuning data, TinyChat, is licensed CC-BY-NC-4.0, and its card's terms carry over to this model. The base model, pk_base_v1, has no non-commercial data and is MIT-licensed.

Files

file	format	size	sha256
pk_sft_v1.gguf	GGUF, F32 (the primary file)	441.1 MB	4d53c7ff236ecba3a2b34cd2d4769f08bdf0408bbc3655afa44b9ddc70929fdd
pk_sft_v1.safetensors	paperkiln's own tensor names, F32	390.2 MB	eae61755dc6625eb87c54ce3152774596ed4299c65c0462ed3831b0f9d7ee80e
spec.json	the paperkiln fine-tuning spec
result.json	the run's summary

SHA256SUMS lists every file's hash. The GGUF carries the tokeniser, the chat template (as paperkiln's paperkiln.chat_template record and as the standard Jinja tokenizer.chat_template) and paperkiln's IR (a JSON description of the network, paperkiln.ir, version 1). The safetensors file is for paperkiln (mtstudio export spec.json --from pk_sft_v1.safetensors rebuilds the GGUF); the embedding and output matrix are tied in it and written as two tensors in the GGUF (llama.cpp reports 110.1M parameters for that reason).

No quantised file is published: the Q8_0 export's quantisation error was not measured to paperkiln's gate, and no 4-bit file of this model has been made.

Architecture

The same as pk_base_v1: Llama-style decoder, 97,536,768 parameters (output matrix tied to the token embedding), 12 layers, width 768, 12 attention heads of 64 (no grouped queries), SwiGLU feed-forward of width 2,048, RMSNorm (epsilon 1e-5, pre-norm), RoPE (theta 10,000), no biases, 512-token context, 16,384-entry vocabulary.

Tokeniser

Its own byte-level BPE with 16,384 entries in the Llama 3 scheme (Llama 3's pre-tokeniser regex, GPT-2's byte-to-unicode mapping), trained on 205 MB of pk_base_v1's pretraining sources. Ids 0-3 are control tokens: <unk>, <|endoftext|>, <|user|>, <|assistant|>. No BOS token is added. In the GGUF the tokeniser is tokenizer.ggml.model = gpt2, tokenizer.ggml.pre = llama-bpe, so llama.cpp and Ollama tokenise text themselves: llama.cpp b6000's tokeniser gave the same ids as paperkiln's on 794 texts (84,934 tokens). It is not Llama 3's vocabulary; only the scheme is shared.

Chat template

<|user|>{user message}<|assistant|>{reply}<|endoftext|><|user|>{user message}<|assistant|>

No BOS and no whitespace between turns. A system message, if given, is placed at the start of the first user message followed by a blank line (<|user|>{system}\n\n{user message}); the model was not trained with system messages. Generation stops at <|endoftext|> (and engines also stop at <|user|> or <|assistant|>). The Jinja template in the GGUF renders exactly this; llama.cpp's --jinja rendering equals paperkiln's on 60 conversations (single turn, multi-turn, with a system message).

How to run it

ember.cpp (C++ inference engine, CPU or CUDA; /chat and an OpenAI-compatible API):

git clone https://github.com/JohnnyTeutonic/ember.cpp.git && cd ember.cpp
mkdir -p build && cd build && cmake .. -DCMAKE_BUILD_TYPE=Release && make -j && cd ..
hf download johnnyteutonic/pk_sft_v1 pk_sft_v1.gguf --local-dir models
./build/tinyllama_server models/pk_sft_v1.gguf 8080 127.0.0.1 ./www 0   # 0: CPU; -1: all layers on the GPU
curl http://127.0.0.1:8080/v1/chat/completions -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "user", "content": "What is the capital of France?"}]}'

Open http://127.0.0.1:8080 for a chat page.

llama.cpp (tested at b6000):

llama-server -m pk_sft_v1.gguf --jinja -c 512 --port 8080

Ollama (tested with 0.40.11), with this Modelfile beside the GGUF:

FROM ./pk_sft_v1.gguf
TEMPLATE """{{ $first := true }}{{ range .Messages }}{{ if eq .Role "user" }}<|user|>{{ if and $first $.System }}{{ $.System }}

{{ end }}{{ $first = false }}{{ .Content }}{{ else if eq .Role "assistant" }}<|assistant|>{{ .Content }}<|endoftext|>{{ end }}{{ end }}<|assistant|>"""
PARAMETER stop "<|endoftext|>"
PARAMETER stop "<|user|>"
PARAMETER stop "<|assistant|>"

ollama create pk-sft-v1 -f Modelfile && ollama run pk-sft-v1

transformers: AutoModelForCausalLM.from_pretrained("johnnyteutonic/pk_sft_v1", gguf_file="pk_sft_v1.gguf") loads the weights (the benchmark scores below were computed this way, with token ids from paperkiln's tokeniser, tools/llama3_bpe.py).

Engine agreement, greedy decoding on 40 fixed prompts (60 new tokens): ember.cpp on CPU and on CUDA, llama.cpp b6000 (/v1/chat/completions) and Ollama 0.40.11 (/api/chat) each give paperkiln's own sampler (mtstudio chat) identical replies and prompt token counts, 40/40.

Training data

Fine-tuning (this model): 313,606 conversations with 463,695 assistant replies, one row per conversation.

source	revision	licence	conversations	assistant replies
smol-smoltalk, subsets smol-magpie-ultra-short, openhermes-50k and everyday-conversations	f73fe857	Apache-2.0	284,442 (90.7%)	337,415 (72.8%)
TinyChat (tinychat.txt, first 20 MiB)	1a19729d	CC-BY-NC-4.0	27,163 (8.7%)	118,442 (25.5%)
everyday-conversations-llama3.1-2k, train_sft	14f54321	Apache-2.0	2,001 (0.6%)	7,838 (1.7%)

Licences are those on each dataset's card at the revision used. Only alternating user and assistant turns from the first user turn were kept (system messages dropped), and every source was checked against paperkiln's own frozen evaluation sets (n-gram, span and prompt rules), which dropped 372 conversations; the data were not checked against the public benchmarks below. Replies that would not fit the 512-token context with their history were dropped, so long conversations keep only their first replies. The fine-tuning data are machine-generated: smol-magpie-ultra-short and everyday-conversations by Llama 3.1, openhermes-50k drawn from OpenHermes 2.5 (largely GPT-4 output), and TinyChat by GPT-4o-mini (per its card).

Pretraining (inherited from pk_base_v1, MIT): 983M tokens of FineWeb-Edu (ODC-By 1.0; 70.0% of the bytes), UltraChat 200k (MIT; 17.5%), SODA (CC-BY-4.0; 12.3%), OASST1/OASST2 (Apache-2.0; 0.16%) and everyday-conversations-llama3.1-2k (Apache-2.0; 0.03%); details on the pk_base_v1 card.

Training

• Initialised from pk_base_v1 (pk_base_v1.safetensors, sha256 1251f204...).
• 18,620 steps of 16 conversations (about one pass over the 297,926 training conversations), 80.2M tokens in the training rows, no conversation over the 512-token context. The loss is on the assistant replies only (masked), every reply of each conversation.
• AdamW, peak learning rate 1e-4, 200 warmup steps, cosine decay to 10%, seed 1, BF16 mixed precision, fused (flash) attention.
• One NVIDIA L4 GPU (RunPod), about 17,100 tokens per second: 2.47 GPU-hours.
• Held-out loss 1.508 nats per token (512 of 15,680 held-out conversations), still falling at the end.

Evaluation

Simple-QA probe (paperkiln's corpus/eval/simple_qa_v1.jsonl: 60 short questions, each asked as a single user turn and answered greedily; a reply counts when it contains an accepted answer, so a reply that lists the right answer among wrong ones counts):

	overall	fact (25)	science (15)	arithmetic (10)	follow the turn (10)
pk_sft_v1	21/60	11/25	5/15	0/10	5/10

It answers common facts (Paris, Rome, London, the Pacific, Jupiter) and gets others wrong in fluent sentences ("The capital of Japan is ... located in the country of Hokkaido"); it scored 0/10 on arithmetic.

Standard benchmarks (zero-shot log-likelihood forced choice in lm-evaluation-harness's formats, with paperkiln's harness, tools/bench_eval.py; BLiMP uses the first 100 pairs of each of its 67 paradigms):

model	params	HellaSwag (acc_norm)	ARC-Easy	ARC-Challenge (acc_norm)	PIQA	WinoGrande	LAMBADA	BLiMP
chance		0.250	0.250	0.250	0.500	0.500	0	0.500
pk_base_v1	97.5M	0.299	0.437	0.253	0.612	0.521	0.252	0.755
pk_sft_v1	97.5M	0.295	0.436	0.243	0.605	0.511	0.243	0.740
Pythia-160m	162M	0.306	0.447	0.244	0.618	0.525	0.357	0.783
SmolLM2-135M	135M	0.436	0.644	0.291	0.679	0.534	0.436	0.803

Items: HellaSwag 10,042, ARC-Easy 2,376, ARC-Challenge 1,172, PIQA 1,838, WinoGrande 1,267, LAMBADA 5,153, BLiMP 6,700. The fine-tuning moves every score by 0.001-0.015 from the base model's, inside the 95% intervals. ARC-Challenge and WinoGrande are at chance. LAMBADA target-word perplexity 109.3 (pk_base_v1 89.3).

Instruction following (paperkiln's IF-tiny, 574 prompts with code-checked instructions such as word limits or a required word): macro compliance 0.123, against 0.062 for the same prompts without the instruction; SmolLM2-135M-Instruct 0.276 and Qwen2.5-0.5B-Instruct 0.515 on the same harness.

Limitations

• A 97.5M-parameter model with a 512-token context; conversations longer than that are cut.
• Frequent factual errors stated fluently (a spider "has 4 legs"; Rome "in the Italian Alps"); no arithmetic.
• Long answers loop: on 100 open-ended UltraChat test prompts, 56% of sampled replies (64 tokens, temperature 0.7) were flagged as repeating themselves.
• English only.
• No preference tuning and no safety tuning: it has no refusal training and can produce false, offensive or unsafe content.
• Not evaluated for bias or toxicity.
• Trained on machine-generated conversations, so it imitates other models' styles and errors.

Licence

CC-BY-NC-4.0, because the fine-tuning data include TinyChat (CC-BY-NC-4.0); use is non-commercial. The other sources' licences (Apache-2.0, MIT, ODC-By 1.0, CC-BY-4.0) permit this. Attribution:

• smol-smoltalk and everyday-conversations-llama3.1-2k: Hugging Face (HuggingFaceTB), from the SmolLM2 work (Allal et al., 2025); Apache-2.0.
• TinyChat: starhopp3r (Hugging Face); CC-BY-NC-4.0.
• Pretraining sources (via pk_base_v1): FineWeb-Edu (Penedo et al., 2024; ODC-By 1.0, subject to the Common Crawl terms of use), SODA (Kim et al., EMNLP 2023; CC-BY-4.0), UltraChat (Ding et al., 2023; MIT), OpenAssistant Conversations (Köpf et al., 2023; Apache-2.0).

Users should also check the terms of the models that generated the synthetic sources.