rostlabs/rost-1b-instruct-GGUF

rost-1b-instruct, GGUF

These files will not load in Ollama, LM Studio, or a stock llama.cpp build

rost is not an architecture llama.cpp knows. llama.cpp compiles architectures in rather than loading them dynamically, so support has to be present in the binary. It is not upstream yet, and Ollama and LM Studio bundle their own llama.cpp. A stock build will refuse these files with unknown model architecture: 'rost'.

A patched build is required. Instructions are below.

Quantisations of rostlabs/rost-1b-instruct, a 1.4B bilingual Romanian/English model.

Files

file	size	bits/weight	notes
rost-1b-instruct-Q4_K_M.gguf	791 MiB	4.79	the one to start with
rost-1b-instruct-Q6_K.gguf	1.08 GiB	6.56
rost-1b-instruct-Q8_0.gguf	1.40 GiB	8.50
rost-1b-instruct-BF16.gguf	2.64 GiB	16.00	the training precision; no quantisation loss

Measured on this checkpoint, generating with twelve CPU threads and on one RTX 5070:

	RTX 5070	12 CPU threads
Q4_K_M	489.7 tok/s	60.6 tok/s
F32 (for scale)	170.1 tok/s	14.3 tok/s

Building a llama.cpp that can read these

There is a fork with the architecture already in it. Three lines:

git clone -b master https://github.com/rostlabs/llama.cpp && cd llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build -j

Its master is upstream llama.cpp master plus a single commit adding the architecture. Metal is included automatically on macOS; add -DGGML_CUDA=ON for NVIDIA.

The architecture is one commit; git log -1 in that clone shows what it adds and why.

Then:

./build/bin/llama-cli -m rost-1b-instruct-Q4_K_M.gguf \
  --repeat-penalty 1.1 -p "Care este capitala României?"

Sampling

Use a repetition penalty of 1.1. It is not optional decoration on this model. Measured over 198 generations of 400 tokens on prompts chosen to provoke the failure:

decoding	replies that looped	worst repeated 6-gram
greedy	50%	44x
temperature 0.2, no penalty	44%	44x
temperature 0.6, no penalty	22%	10x
temperature 0.6, penalty 1.1	0%	2x

Stronger penalties suppress repetition further and cost accuracy: at 1.3 the model stops looping and starts inventing etymologies. 1.1 is the mildest setting that works.

The transformers repository ships these defaults in generation_config.json; llama-cli needs --repeat-penalty 1.1 passed explicitly.

How faithful are these files?

The port is checked against a second implementation of the forward pass, written from the safetensors, which agrees with the published model to 1.7e-05:

prefill, stage by stage	5.8e-06 to 1.8e-04
single-token decode, top-8 probabilities	4.9e-05
after a cache rollback	8.0e-05
two concurrent sequences	1.8e-05 to 1.5e-04
tokenisation	exact on 8 of 8 strings, including whitespace runs, digit groups and emoji
CUDA	top-8 order matches, 3.8e-04 prefill and 2.7e-04 decode
Metal	top-8 order matches, 4.3e-05 prefill and 1.2e-04 decode

On an M3 the Q4_K_M file runs at 1435 t/s prompt and 77 t/s generation.

At BF16 the greedy output matches the reference exactly. At Q4_K_M a token can differ, which is ordinary quantisation.

A note on file revisions

These files were regenerated on 2026-08-20. Two things the checkpoint does unusually — a transposed rotary convention, and a QK norm with no learnable weight — turned out to fold into the weights, so the architecture now uses llama.cpp's ordinary rope and its existing q_norm/k_norm path instead of carrying special cases. The maths is exact, and the outputs are unchanged.

The file format changed as a result: an earlier download will not load in a current build, and these files will not load in a build from before that change. If you took a copy on the day of publication, take it again.

Limitations

Everything in the main model card applies: it confabulates confidently, its reasoning is near chance, the context is 4,096 tokens, and no safety tuning of any kind has been applied. Read that page before using this.