nano-kyrgyz

A small Kyrgyz language model, written from scratch in PyTorch and trained on a laptop. 87M parameters. Not a fine-tune of anything — every component is original (source repository).

---

Start here

I want to…	Download	Size
chat in Kyrgyz	nano-kyrgyz-0.7-chat-q8_0.gguf	93 MB
continue Kyrgyz text	nano-kyrgyz-0.7-q8_0.gguf	93 MB
the smallest thing that works	nano-kyrgyz-0.7-chat-Q4_K_M.gguf	54 MB

# chat
llama-cli -m nano-kyrgyz-0.7-chat-q8_0.gguf --jinja -c 512 --temp 0.7 --top-k 40

# continue a sentence you start — give it the BEGINNING of one, not a greeting
llama-completion -m nano-kyrgyz-0.7-q8_0.gguf -c 512 --temp 0.7 --top-k 40 \
  --no-conversation -n 200 -p "Кыргыз Республикасынын"

The base model is a continuator, not a chatbot. Prompt it with an unfinished sentence. Greet it and it will correctly decide the text is over and return nothing — at greedy that happens on every greeting-shaped prompt. Use the chat model to chat.

In LM Studio: search DJKG/nano-kyrgyz, set context to 512, temperature 0.7, top-k 40, and turn off "Chat with Files". No prompt-template setup — every file ships its own. Those sampling settings matter more on this release than on the last one: on the v0.7 base greedy decoding degenerates on 35.4% of held-out prompts and temperature 0.7 brings that to 7.4%. The same two numbers on v0.6 were 29.3% and 2.7%, so if you serve the base greedily you will notice the difference — see the regression table below (assets/eval_history.json, 512 prompts, 1200-char budget).

Older filenames carry the training-run number, not the release number — -v5- is the v0.3 release, -v3- is v0.2. From v0.4 the filename is the release.

---

What it can and cannot do

Works well: greetings, saying what it is, repeating and continuing short text you give it, answering questions about a document you paste in — including saying which field the document does not have — and declining to invent an answer.

Салам!                              → Салам! Суроо бериңиз.
Мага жардам бер                     → Албетте. Мен текст менен иштейм: кайталайм,
                                       улантам, тамгаларды котором, документтеги
                                       талаалар боюнча суроого жооп берем. Кайсынысы керек?
Ыр жаз                              → Кечиресиз, ыр же аңгеме жазуу менин колумдан
                                       келбейт. Сиз жазган текстти кайталай же улантып
                                       бере алам.
Кайталап жаз: Мен мектепке барам.   → Мен мектепке барам.
Документ: Сумма: 538,29 сом
Суроо: Сумма канча?                 → 538,29 сом
Кыргызстандын калкы канча?          → Кечиресиз, мен так маалыматты билбейм.
                                       Жаңылыш айтып койбоюн.

(Every line above is this model's actual greedy output, not an illustration. In v0.2 chat the first three came back as Салам!, the words "I cannot translate", and your own prompt echoed back.)

Does not work. This is an 87M-parameter model — it holds language, not facts:

• Facts, arithmetic, reasoning, translation. It is trained to say so rather than guess. It understands simple instructions and simple language; it does not know simple facts, and a small model answering «I don't know» is the design working.
• Kyrgyz only. Russian and English input get a polite decline, not an answer.
• It cannot write for you. Ask for a poem, a story or an essay and it will say so. It continues a sentence you start; it does not compose.
• Case conversion is unreliable, but it improves with scale. A byte-level BPE model sees бишкек and БИШКЕК as unrelated tokens, so the operand's identity does not survive re-segmentation. An earlier version of this card called that "a property of the tokenizer, not of how much it was trained" — measured across 180 held-out words, that was wrong: doubling the parameters took it from 37% to 53%, and from 7/30 to 13/30 on eight-letter words. It is still the weakest thing here; it is not a hard ceiling.
• Number extraction tends to return the first number rather than all of them.
• No RLHF and no safety tuning.

Measured rather than guessed. scripts/chat_sheet_first5.py runs 40 prompts written from what a first-time visitor actually types — none of them anywhere in the training data, and none of them ever used to pick a checkpoint. Mechanically broken answers (echoing your words back, looping, never stopping, answering as if about a pasted document, or declining in Kyrgyz because it thinks you are not speaking Kyrgyz):

	broken	stops	verbatim copy, exact
v0.2 chat	21/40	100%	85%
nano-kyrgyz-0.7-chat	0/40	100%	86%

Hand-scored, about 30 of 40 answers are genuinely good, against 13 for v0.2 chat. The mechanical number is the honest floor, not the ceiling — it counts the ways a model looks broken, not the ways it can be wrong, and the two are not the same thing.

The copy column is there because the first v0.4 build proved the point: it lost 13 points of verbatim-copy fidelity and every mechanical flag stayed clean through it, because a wrong copy is fluent, terminates and repeats nothing. Copy is measured separately now, on 280 held-out fragments across seven lengths (scripts/chat_task_gates.py), and a release that drops below the last one has to say so out loud.

---

The numbers

Bits per character on 3,000 held-out passages of raw text, produced by scripts/eval_bpc_text.py — never typed by hand.

Why not one val split, as previous releases quoted? This release retrains the tokenizer, and the usual harness reads a pre-tokenised file — so its bits-per-char is only comparable between models that share a vocabulary. Scoring raw text with each model's own tokenizer, and dividing by characters, is comparable across that change. Dividing by tokens would not be: that is the bug that put four releases of wrong orthography numbers on this page.

What changed in v0.7: the corpus. Every earlier release trained on ~95% web crawl. This one adds 245M characters of edited Kyrgyz literature — novels, the Manas epic, fairy tales, proverbs, textbooks — digitised from public libraries, 62M of it recovered by OCR from books that exist only as page scans. The tokenizer was retrained on the result.

	v0.6 base	v0.7 base
parameters	86.9M	86.9M
training tokens	1835.0M (4.22 epochs)	1835.0M (3.80 epochs)
bits/char, literary text ↓	1.5780	1.3817
bits/char, web text ↓	1.0628	1.0731
characters per token, literary ↑	3.299	3.444
orthography ң ө ү ↑	98.7%	99.0%

Literary bits-per-char improves by 0.196. For scale, doubling the parameter count in v0.6 — three days of compute — moved it 0.086. Web text costs 0.010 bpc, which is flat. The gap between how well the model handles edited prose and how well it handles web text narrows from 0.515 to 0.309.

It splits into two effects and both are real: the retrained tokenizer needs 4.4% fewer tokens for the same literary text, and the model is 8.6% less surprised per token on top of that.

The base got worse at free-running generation, and that is not hidden

Measured over 512 held-out prompts at temperature 0.7, against the v0.6 base:

	v0.6 base	v0.7 base
stops on its own ↑	63.1%	55.9%	worse, p = 0.022
degenerates ↓	29.3%	35.4%	worse, p = 0.045
repeated 4-grams ↓	34.5%	43.7%	worse, p = 0.003

All three are statistically significant and all three point the same way. Literature is long-form continuous prose and epic verse is formulaic and repetitive; a corpus with 245M characters of it makes a better model of Kyrgyz and a more repetitive continuator.

This does not carry into the chat model, which is the file most people want: the SFT teaches the answer-then-stop format, and nano-kyrgyz-0.7-chat stops on its own 100% of the time with 0/40 mechanically broken answers on a held-out sheet it was never selected on. If you are using the base as a free-running continuator, prefer v0.6; for everything else v0.7 is the better model.

Retracted: the orthography regression this card first reported. The v0.6 release note said orthography dropped from 93.3% to 92.4%, called it "measured, not excused" and "not yet understood". It is now understood, and it was not a regression — it was a scoring bug in this repo, and the number above is the corrected one.

eval_orthography compared the per-token mean log-likelihood of the true and re-spelled sentences. But ө→о, ү→у and ң→н each swap one character for one, so the two strings always have identical character length — the only thing that division could vary is the BPE segmentation, and it varies in one direction: Russifying a word shatters the Kyrgyz merges the tokenizer learned, so the correct spelling tokenises 1.74 tokens shorter on average, in 79% of pairs. The shipped test was therefore "does the true spelling win by more than that many tokens' worth of average surprisal" — a threshold set by the tokenizer, not by spelling. 86–92% of every error this metric ever reported were pairs where the model assigned higher probability to the correct Kyrgyz spelling and the length term overturned it.

Comparing total sequence log-likelihood instead — the tokenizer-free question, and identical to a per-character comparison since character count is fixed — on 3,000 held-out pairs per letter rather than ~192:

	v0.2	v0.3	v0.5	v0.6
orthography, corrected	99.0%	98.8%	98.9%	98.8%
as previously published	93.8%	94.5%	93.3%	92.4%

No trend and no regression: every checkpoint sits between 98.8% and 99.0%. The narrated slide was the tokenizer term plus a sample too small to see anything under about four points — 200 pairs per letter, drawn from a fixed prefix covering 3.9% of the split, with no seed to vary.

And on the axis a reader actually experiences, v0.6 was the cleanest checkpoint measured. This was not re-measured for v0.7, so it is reported as the v0.6 result it is. Across 600 held-out prompts it wrote a Kyrgyz word with ң/ө/ү flattened in 0.089% of opportunities, against v0.5's 0.320% (Fisher exact p = 0.011) — and against 0.481% in the matched human text it learned from.

Orthography is the metric this project cares about most: held-out sentences are re-spelled with the confusable letter (ө→о, ү→у, ң→н) and the model must prefer the true spelling. These are the exact letters Russian-trained and OCR stacks collapse. Per letter, v0.2: ө/о 97.0%, ү/у 94.5%, ң/н 89.8%.

The chat model is instruction-tuned from ckpt_v8.pt, the base released above. It is the first chat model here that is clean on both held-out sheets, and it copies text back to you more accurately than any predecessor — 83.2% exact on 280 held-out fragments, better in every length band.

A correction, found the day after release. This paragraph first claimed the chat model extracted multiple numbers from a sentence "more than twice as often, 22.5% → 47.5%", and credited about 12 of those points to the larger base. That was a measurement error, not a result. chat_task_gates.py defined a number with one regex and the SFT trained on another: the trainer treats 309,5 as a single decimal, the gate split it into 309 and 5, so the model was marked wrong for producing exactly what it was taught — the same shape of bug as the orthography metric above. With both sides sharing one definition:

	1 number	2+ numbers	verbatim copy	n
v0.4.1 (42M)	85.0%	37.5%	78.6%	40 / 280
v0.5 (42M)	90.0%	37.5%	77.9%	40 / 280
v0.6 (87M)	85.0%	37.5%	83.2%	40 / 280
v0.6 chat, re-measured	92.0%	45.0%	86.4%	200 / 1,050
v0.7 chat	96.5%	46.0%	88.2%	200 / 1,050

Correction to this table. Earlier releases reported multi-number extraction as "37.5% in every release, never moved". That figure was measured on 40 sentences, where the gate cannot resolve a difference smaller than about 15 points. Re-measured on 200, the same v0.6 checkpoint scores 45.0% — so the flat line was partly the sample size, not the model. The honest statement is that multi-number extraction is somewhere in the mid-40s and has not clearly improved; none of the v0.6→v0.7 differences in this table are statistically significant on their own. If you paste a sentence with several numbers in it, expect the model to miss some — that is the honest state of it.

---

Files

file	what it is	size
nano-kyrgyz-0.7-chat-q8_0.gguf	v0.7 chat — start here	93 MB
nano-kyrgyz-0.7-chat-Q4_K_M.gguf / -f32.gguf	v0.7 chat, 4-bit / unquantised	54 MB / 348 MB
nano-kyrgyz-0.7-q8_0.gguf	v0.7 base, 86.9M — start here for text	93 MB
nano-kyrgyz-0.7-Q4_K_M.gguf / -f32.gguf	v0.7 base, 4-bit / unquantised	54 MB / 348 MB
nano-kyrgyz-0.5-*.gguf	v0.5 base, 42.2M — half the size, and it shows	93 MB / 54 MB / 348 MB
nano-kyrgyz-0.4.1-chat-*.gguf, nano-kyrgyz-v3-*.gguf	previous releases	—
nano-kyrgyz-*.gguf (no version)	v0.1, 5.7M char-level — reference only	6.1–22.9 MB
vision/*.pt	document pipeline checkpoints (PyTorch, run from the source repo)	5.5–40 MB

From v0.4 the filename carries the release number. Older files carry the training-run number instead — -v5- is the v0.3 release and -v3- is v0.2 — which is the confusion the new scheme exists to end.

Also runs under Ollama and any OpenAI-compatible client via LM Studio's server on :1234. Modelfiles are in the source repo — note the suffix, Modelfile.chat without one is the old v0.1 model:

ollama create nano-kyrgyz-chat -f export/Modelfile.nano-kyrgyz-v3-chat
ollama run nano-kyrgyz-chat "Салам!"

---

Document understanding

The source repo ships a from-scratch document pipeline — YOLO11n layout detection → line crops → a 10M-parameter nano-VLM OCR → field linking → JSON — and vision/ask.py, which hands that JSON to the chat model so you can photograph a Kyrgyz form and ask about it in Kyrgyz. OCR: 19.7% CER, 53.6% digit error, 38.6% CER on field values. A confidence gate withholds lines it could not read rather than guessing. Still measured on synthetic pages only.

---

How it was built

Trained on an Apple M5 Pro, on 588.8M characters of Kyrgyz (Leipzig Wikipedia + NewsCrawl + CC-100 + MADLAD-400), cleaned through NFC normalisation, homoglyph repair, a Cyrillic-ratio filter and global dedup. Byte-level BPE, 8,192 tokens. bf16, EMA, WSD schedule, Muon on hidden matrices + AdamW elsewhere. Total cost: $0.

Measured, not assumed: Muon beat AdamW by −0.126 val loss at equal steps; QK-norm earned its cost; z-loss slightly hurt at this scale; MQA matched GQA at half the KV cache.

Shipped as architecture qwen3. These weights share nothing with any Qwen model. The model was deliberately built from the same components (RMSNorm, NEOX RoPE, GQA, per-head QK-norm before RoPE, SwiGLU) so llama.cpp's existing graph runs it without a patch. The bar for calling that honest is token parity, not "it runs": at --temp 0, llama.cpp output is character-identical to the reference PyTorch implementation.

Four bugs worth reading about — a frequency cutoff that silently deleted the letter Ң from the alphabet; 482 archaic ѳ (U+0473) homoglyphs hiding in 0.01% of the corpus; an add_bos_token flag that fed a never-pretrained token to the base model and turned the project's own headline example into a repetition loop; and one data-prep constant that taught a model not to stop, costing it the release despite the best bpc of any run. Each is written up in full in the source repository — they are the point of the project.

---

Licence

Weights Apache 2.0. Training data comes from the Leipzig Corpora Collection (CC BY-NC 4.0), CC-100 and MADLAD-400; the corpus licence is non-commercial and the weights are released as research output — evaluate your own use accordingly. Handwriting glyphs from Kyrgyz MNIST (CC BY-NC 4.0). No Qwen weights, data or code were used.

Built by Bakyt Djumabaev.