pk_base_v1

pk_base_v1 is a 97.5M-parameter decoder-only language model in the Llama architecture, trained from scratch by Jonathan Reich with paperkiln, a C++ training framework with hand-written backward passes. It is a base model: it continues text and has had no instruction tuning. Its chat version is pk_sft_v1.

Files

file	format	size	sha256
pk_base_v1.gguf	GGUF, F32 (the primary file)	441.1 MB	510d288cf10ead4d3902247e3d898b6fa39e1d31d7465d3a8b6820761d6c9178
pk_base_v1_q4_k_m.gguf	GGUF, Q4_K_M (73 Q4_K, 13 Q6_K, 25 F32 tensors)	69.2 MB	78b3482f3ba3bc050ba1c6d019803b898f6c480d5773e2b75a552613d2e8272f
pk_base_v1_q4_k_s.gguf	GGUF, Q4_K_S (80 Q4_K, 5 Q5_K, 1 Q6_K, 25 F32 tensors)	66.4 MB	7980aa549815a20c5626a51cf5ac4249be08bfa4c84b0366d9f026837d6105ca
pk_base_v1.safetensors	paperkiln's own tensor names, F32	390.2 MB	1251f204342299c54a30d15601976bf5dc4c44611d42bbb988c2e2ed25a686fd
spec.json	the paperkiln training spec
result.json	the run's summary

SHA256SUMS lists every file's hash. Each GGUF carries the tokeniser and paperkiln's IR (a JSON description of the network: paperkiln.ir). The F32 file was exported before paperkiln's conversation schema existed, so it has no chat template and carries IR version 0; the two 4-bit files, exported later from the same weights, carry IR version 1 and paperkiln's chat template (the same as pk_sft_v1's). The safetensors file is for paperkiln (mtstudio export spec.json --from pk_base_v1.safetensors rebuilds the GGUF); the embedding and output matrix are tied in it and written as two tensors in the GGUFs (llama.cpp reports 110.1M parameters for that reason).

Architecture

family	Llama-style decoder (general.architecture = llama)
parameters	97,536,768 (output matrix tied to the token embedding)
layers	12
width	768
attention	12 heads of 64, full multi-head (no grouped queries), causal
feed-forward	SwiGLU, inner width 2,048
normalisation	RMSNorm, epsilon 1e-5, pre-norm
positions	RoPE, theta 10,000, every head
context	512 tokens
vocabulary	16,384
biases	none

Tokeniser

Its own byte-level BPE with 16,384 entries in the Llama 3 scheme: the Llama 3 pre-tokeniser regex, GPT-2's byte-to-unicode mapping, and merges trained with Hugging Face tokenizers on 205 MB of the training sources (FineWeb-Edu 146.8 MB, UltraChat 26.2 MB, SODA 21.0 MB, OASST 10.5 MB, everyday-conversations 1.9 MB). Ids 0-3 are control tokens: <unk>, <|endoftext|>, <|user|>, <|assistant|>. No BOS token is added; <|endoftext|> separates documents. In the GGUF the tokeniser is tokenizer.ggml.model = gpt2, tokenizer.ggml.pre = llama-bpe, so llama.cpp tokenises text itself: llama.cpp b6000's tokeniser gave the same ids as paperkiln's on 794 texts (84,934 tokens). It is not Llama 3's vocabulary; only the scheme is shared.

How to run it

ember.cpp (C++ inference engine, CPU or CUDA):

git clone https://github.com/JohnnyTeutonic/ember.cpp.git && cd ember.cpp
mkdir -p build && cd build && cmake .. -DCMAKE_BUILD_TYPE=Release && make -j && cd ..
hf download johnnyteutonic/pk_base_v1 pk_base_v1.gguf --local-dir models
./build/tinyllama_server models/pk_base_v1.gguf 8080 127.0.0.1 ./www 0   # 0: CPU; -1: all layers on the GPU

llama.cpp (tested at b6000):

llama-server -m pk_base_v1.gguf -c 512 --port 8080
curl http://127.0.0.1:8080/completion -d '{"prompt": "The water cycle is", "n_predict": 60, "temperature": 0}'

transformers: AutoModelForCausalLM.from_pretrained("johnnyteutonic/pk_base_v1", gguf_file="pk_base_v1.gguf") loads the weights (the benchmark scores below were computed this way, with token ids from paperkiln's tokeniser, tools/llama3_bpe.py).

Engine agreement: greedy decoding on 40 fixed prompts (60 new tokens) gives identical replies, token counts and stop reasons in ember.cpp (CPU) and paperkiln's own sampler (mtstudio chat), 40/40, and in llama.cpp b6000 on the F32 file, 40/40 (with an F32 or the default F16 KV cache).

Training data

source	revision	licence	documents used	share of training bytes
FineWeb-Edu, sample-10BT shard 000_00000	87f09149	ODC-By 1.0 (and the Common Crawl terms of use)	613,309	70.0%
UltraChat 200k, train_sft	8049631c	MIT	126,365	17.5%
SODA, train	fdc848ab	CC-BY-4.0	706,117	12.3%
OASST2 and the OASST1 trees not in OASST2, English, best-ranked branch per tree	179dd21f, fdf72ae0	Apache-2.0	3,328	0.16%
everyday-conversations-llama3.1-2k, train_sft	14f54321	Apache-2.0	1,394	0.03%

Licences are those on each dataset's card at the revision used. The dialogue sources are rendered as text in the chat format (<|user|>...<|assistant|>...<|endoftext|>), so conversation is part of pretraining: 27% of the bytes in the first 85% of training and 47% in the last 15% (an annealing tail with its own mix, read from step 102,000). Dialogues were deduplicated within each source, and a check against paperkiln's own frozen evaluation sets (n-gram and span rules) dropped 4,117 of them; the data were not checked against the public benchmarks below. Much of the dialogue text is machine-generated: UltraChat by ChatGPT, SODA by an OpenAI GPT-3.5 model, everyday-conversations by Llama 3.1; OASST is human-written.

The mix is 4.20 GB of text, 958.5M tokens (816.0M in the main part, 142.5M in the tail).

Training

• 120,000 steps of 16 sequences of 512 tokens: 983M tokens, about one pass over the mix.
• AdamW, peak learning rate 6e-4, 2,000 warmup steps, cosine decay to 10%, gradient clipping at 1.0, seed 1, FP32 (no mixed precision).
• One NVIDIA L4 GPU (Google Colab), about 9,600 tokens per second: about 28 GPU-hours of training.
• Held-out loss 2.676 nats per token (40.8M held-out tokens of the mix).

Evaluation

Zero-shot log-likelihood forced choice in lm-evaluation-harness's formats, with paperkiln's harness (tools/bench_eval.py; 95% Wilson intervals in the source table). BLiMP uses the first 100 pairs of each of its 67 paradigms. Pythia-160m and SmolLM2-135M were scored with the same harness for reference.

model	params	HellaSwag (acc_norm)	ARC-Easy	ARC-Challenge (acc_norm)	PIQA	WinoGrande	LAMBADA	BLiMP
chance		0.250	0.250	0.250	0.500	0.500	0	0.500
pk_base_v1	97.5M	0.299	0.437	0.253	0.612	0.521	0.252	0.755
pk_sft_v1	97.5M	0.295	0.436	0.243	0.605	0.511	0.243	0.740
Pythia-160m	162M	0.306	0.447	0.244	0.618	0.525	0.357	0.783
SmolLM2-135M	135M	0.436	0.644	0.291	0.679	0.534	0.436	0.803

Items: HellaSwag 10,042, ARC-Easy 2,376, ARC-Challenge 1,172, PIQA 1,838, WinoGrande 1,267, LAMBADA 5,153, BLiMP 6,700. LAMBADA target-word perplexity: pk_base_v1 89.3, Pythia-160m 35.9, SmolLM2-135M 17.8. ARC-Challenge and WinoGrande are at chance for this model. The same harness gives SmolLM2-135M within 2.1 points of its own model card (HellaSwag 42.1, PIQA 68.4, WinoGrande 51.3 there).

The 4-bit files

The Q4_K_M and Q4_K_S files are byte-identical to what llama.cpp b6000's llama-quantize produces from paperkiln's F32 export of the same weights. They passed paperkiln's quantisation gate (the receipts are in ember.cpp, experiments/kquant_2026-10-25/): ember's logits on each file are bit-identical to its logits on the same file dequantised to F32 by gguf-py, on CPU and CUDA, and the quantisation error was measured against the F32 model on the 40 fixed prompts:

file	max abs logit difference (share of the logit range)	first token = F32	60-token greedy reply = F32
Q4_K_M	1.28 (4.6%)	36/40	14/40
Q4_K_S	1.51 (5.5%)	36/40	14/40

Each first-token change goes to the F32 model's second choice, where its margin was 0.003-0.13. These files are for size, not fidelity. llama.cpp's own CPU kernels also quantise the activations to 8 bits for Q4_K, so its greedy replies on Q4_K_M match ember's 29/40 (40/40 once the file is dequantised to F32).

The Q8_0 export of this model is not published: its quantisation error was not measured to the gate's standard.

Limitations

• A 97.5M-parameter model with a 512-token context. It writes fluent English but its knowledge is thin and often wrong; it should not be relied on for facts.
• English only.
• A base model: no instruction tuning, no preference tuning and no safety tuning. It continues whatever text it is given and can produce offensive or false content, including content learnt from web text and from the machine-generated dialogue it was trained on.
• Not evaluated for bias or toxicity.

Licence

The weights are released under the MIT licence, the licence of paperkiln. The training data's licences are listed above; they require attribution, which this card gives:

• FineWeb-Edu: Penedo et al., The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale (2024); ODC-By 1.0, subject to the Common Crawl terms of use.
• SODA: Kim et al., SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualization (EMNLP 2023); CC-BY-4.0.
• UltraChat: Ding et al., Enhancing Chat Language Models by Scaling High-quality Instructional Conversations (2023); the 200k version by Hugging Face H4; MIT.
• OpenAssistant Conversations (OASST1, OASST2): Köpf et al. (2023); Apache-2.0.
• everyday-conversations-llama3.1-2k: Hugging Face (HuggingFaceTB); Apache-2.0.

Users should also check the terms of the models that generated the synthetic sources.