kd13/Type-o1-nano-instruct-GGUF

Type-o1-nano-instruct - GGUF

GGUF quantizations of kd13/Type-o1-nano-instruct, a compact general-purpose instruct model (~0.5B parameters) for everyday assistant use.

Converted with llama.cpp. All quants in this repo are static quants.

Provided quants

Sorted by size, which is not the same as sorted by quality. IQ-quants are often preferable to non-IQ quants of a similar size.

Link	Type	Size/GB	Notes
GGUF	Q3_K_S	0.3
GGUF	Q2_K	0.3
GGUF	IQ4_XS	0.4
GGUF	Q3_K_M	0.4	lower quality
GGUF	Q3_K_L	0.4
GGUF	Q4_K_S	0.4	fast, recommended
GGUF	Q4_K_M	0.4	fast, recommended
GGUF	Q5_K_S	0.4
GGUF	Q5_K_M	0.4
GGUF	Q6_K	0.5	very good quality
GGUF	Q8_0	0.5	fast, best quality
GGUF	f16	1.0	16 bpw, overkill

Which one should I pick?

Q6_K or Q8_0. At 0.5B the entire f16 file is only 1.0 GB, so the disk and memory saved by going lower is measured in a few hundred megabytes while the quality cost is proportionally larger than it would be on a 7B model. A large share of the parameters here sit in the token embedding (vocab 151665), which compresses poorly.

Q4_K_M remains a reasonable floor if you are tightly memory constrained. Q2_K and Q3_K_S are included for completeness rather than as recommendations.

Chat template

This model uses ChatML, the standard Qwen format:

''' <|im_start|>system {system prompt}<|im_end|> <|im_start|>user {message}<|im_end|> <|im_start|>assistant '''

llama.cpp, Ollama, LM Studio, and koboldcpp all recognise this format natively — no extra flags required.

The template carries a built-in default system prompt that is applied when you do not supply one of your own. Pass an explicit system prompt if you want to control the assistant's stated identity and behaviour.

Usage

llama.cpp

llama-completion -m Type-o1-nano-instruct.Q6_K.gguf \
  -sys "You are a helpful assistant." \
  -p "Explain photosynthesis in two sentences."

Recent llama.cpp builds renamed llama-cli to llama-completion; on older builds use llama-cli with the same flags. Add -no-cnv for raw completion with no template.

Server, with an OpenAI-compatible endpoint on port 8080:

llama-server -m Type-o1-nano-instruct.Q6_K.gguf -c 4096 --jinja

Pass --jinja to llama-server if you intend to use tool calling — it makes the server use the model's own template rather than the built-in ChatML handler, which is what renders the <tools> block correctly.

Ollama

ollama run hf.co/kd13/Type-o1-nano-instruct-GGUF:Q6_K

Python

from llama_cpp import Llama

llm = Llama(model_path="Type-o1-nano-instruct.Q6_K.gguf", n_ctx=4096)
out = llm.create_chat_completion(messages=[
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Translate to Hindi: Good morning."},
])
print(out["choices"][0]["message"]["content"])

Tool calling

The template implements Qwen-style function calling. Tool definitions are injected into the system message inside <tools> XML tags, and the model replies with:

''' {"name": "web_search", "arguments": {"query": "..."}} '''

Tool results are returned to the model wrapped in <tool_response> tags. Both llama-server --jinja and Ollama parse this natively and expose it through their OpenAI-compatible tools parameter.

At 0.5B, expect tool-call formatting to be usable but not reliably consistent. Validate the JSON before executing anything, and treat malformed calls as an expected case rather than an error condition.