iapp/OpenThai-SystemOne-GGUF

OpenThai-SystemOne — GGUF

OpenThai-SystemOne is an open Thai + English System One decision model: one forward pass answers typed questions (choice over up to 255 options, ordinal score, yes/no noul) about a text / JSON state with calibrated probabilities, no text generation. It is a Qwen3.5-0.8B text tower (Thai continued pre-training) plus a 256-slot decision head. This repo is a quantization of v0.3 (commit f3709948).

What is quantized: the tower (all Linear layers and the token embeddings, at the GGUF file's level). The 256-slot decision head and the per-type temperatures stay in fp32 (head.safetensors). Quantization therefore only perturbs the hidden state the head reads.

llama.cpp runs the tower and returns its final hidden states (embedding=True, pooling_type=NONE); the included Python client applies the decision head on top, so answers are identical in shape to the PyTorch model.

Files

file	size	accuracy (macro, public / Thai, bf16 original = 74.3 / 80.1)
OpenThai-SystemOne-v0.3-F16.gguf	1517 MB	public / Thai macro 74.2 / 80.2
OpenThai-SystemOne-v0.3-BF16.gguf	1517 MB	same tower weights as F16, stored as bf16 (not scored separately)
OpenThai-SystemOne-v0.3-Q8_0.gguf	812 MB	public / Thai macro 74.1 / 80.0
OpenThai-SystemOne-v0.3-Q6_K.gguf	630 MB	public / Thai macro 74.1 / 80.0
OpenThai-SystemOne-v0.3-Q5_K_M.gguf	578 MB	public / Thai macro 74.1 / 80.1
OpenThai-SystemOne-v0.3-Q4_K_M.gguf	529 MB	public / Thai macro 73.6 / 79.7
OpenThai-SystemOne-v0.3-Q4_0.gguf	501 MB	public / Thai macro 73.8 / 79.5
head.safetensors, head_config.json, tokenizer	1 MB + 20 MB	decision head (fp32) + the tokenizer with the `<
openthai_systemone/	–	the client code (gguf.py = llama.cpp backend)

Usage

pip install llama-cpp-python torch transformers safetensors pydantic   # CMAKE_ARGS="-DGGML_CUDA=on" or "-DGGML_METAL=on" for GPU
huggingface-cli download iapp/OpenThai-SystemOne-GGUF --local-dir openthai-gguf \
    --include "*Q4_K_M.gguf" "head*" "tokenizer*" "openthai_systemone/*"

import sys; sys.path.insert(0, "openthai-gguf")
from openthai_systemone.gguf import GGUFSystemOneClient
c = GGUFSystemOneClient("openthai-gguf/OpenThai-SystemOne-v0.3-Q4_K_M.gguf")   # n_gpu_layers=-1 by default
r = c.system_one("ร้านนี้อาหารอร่อยมาก แต่รอนานเกือบชั่วโมง พนักงานไม่สนใจลูกค้าเลย", {
    "sentiment": {"type": "choice", "instructions": "ความรู้สึกของข้อความ", "criteria": {"บวก": None, "ลบ": None, "กลาง": None}},
    "urgent":    {"type": "noul",   "instructions": "ต้องรีบแก้ไขหรือไม่"},
    "stars":     {"type": "score",  "instructions": "ให้ดาว", "criteria": ["1", "2", "3", "4", "5"]}})
print(r.answers["sentiment"].choice, r.answers["sentiment"].probabilities)

The GGUF alone in llama-cli / llama-server is only the tower: its LM head is the tied input embedding, not the decision head, so generated text is meaningless. Use the client (or read hidden states with --embeddings --pooling none and apply head.safetensors yourself: softmax((h @ W.T + b) / exp(log_temperature[qtype])) over the first k slots + slot 255).

Measured on an H100 (llama-cpp-python 0.3.35, CUDA): ~35 ms per 3-question Thai decision for every level.

Accuracy of Q4_K_M vs the bf16 original (same records, single option order, first 800 per set)

subset	bf16 original	this	Δ
public 13-subset bench
aegis2 (noul)	83.2	83.2	+0.0
boolq (noul)	79.7	78.7	-1.0
civil_comments (noul)	79.0	79.3	+0.3
helpsteer2 (score)	41.6	42.0	+0.4
massive-de-DE (choice)	88.3	86.6	-1.7
massive-en-US (choice)	88.3	88.0	-0.3
multinli (choice)	89.0	87.0	-2.0
paws (noul)	94.0	93.6	-0.4
pubmedqa (choice)	64.0	63.2	-0.8
squad2 (noul)	89.3	86.6	-2.7
summeval-consistency (score)	75.0	75.0	+0.0
summeval-relevance (score)	21.7	21.7	+0.0
vitaminc-dev (choice)	72.5	72.1	-0.3
macro, public 13-subset bench	74.3	73.6	-0.7
Thai held-out / eval sets
banking77 (choice)	59.1	59.2	+0.1
contrastive_th (choice)	80.7	80.4	-0.3
contrastive_th (noul)	83.5	83.1	-0.4
contrastive_th (score)	78.6	78.6	+0.0
massive_th (choice)	90.6	90.1	-0.5
prachathai (choice)	98.3	98.1	-0.2
prachathai (noul)	93.4	93.4	-0.1
sib200_th (choice)	77.9	76.5	-1.5
wisesight (choice)	48.9	49.6	+0.8
wongnai (score)	64.5	63.9	-0.6
xlam_tools (choice)	99.4	99.4	+0.0
xnli_th (choice)	79.8	77.9	-1.9
xnli_th (noul)	86.8	85.5	-1.2
macro, Thai held-out / eval sets	80.1	79.7	-0.5

Notes

• Scores are single-option-order accuracy on the first 800 records of each set (scripts/06_eval.py --limit 800), the same records for the original and the quantization. score subsets report exact level accuracy.
• Base model, data, training and the full benchmark tables: iapp/OpenThai-SystemOne.
• License Apache-2.0 (same as the base). Built by iApp Technology / OpenThaiGPT.