🌐 EN · 🇷🇸 SR · 🇭🇷 HR · 🇧🇦 BS · 🇲🇰 MK · 🇸🇮 SL · 🇦🇱 SQ · 🇲🇪 CNR · 🇧🇬 BG · 🇬🇷 EL · 🇹🇷 TR · 🇷🇴 RO · 🇭🇺 HU
Zora v1.13 — an open, honest LLM for the Balkans & Southeast Europe
зора = "dawn". One to unite them all. — by Sovasoft (ai.in.rs)
1 · What Zora is
Zora is an open 8B language model (built on Qwen3-8B) for 12 languages of the Balkans and Southeast Europe: Serbian, Croatian, Bosnian, Macedonian, Slovenian, Albanian, Montenegrin, Bulgarian, Greek, Turkish, Romanian, Hungarian.
Zora is not built to be the biggest model — it is built to be honest, in-language, and multi-perspective:
- thinks in the target language instead of pivoting through English,
- shows several perspectives on contested topics instead of one national view,
- and above all: admits when it doesn't know instead of inventing facts.
2 · The development story (v1.0 → v1.1 → v1.11 → v1.12 → v1.13)
| Version | Languages | BalkanBench | State |
|---|---|---|---|
| v1.0 | 6 | — | first public release |
| v1.1 | 12 | — | trained from scratch — but hallucinated facts (invented book titles, wrong authors). Never released. |
| v1.11 | 12 | 84/156 | the honest fix: says "I don't know", searches when unsure. #1 Balkan model. |
| v1.12 | 12 | 85/156 | the depth fix: better tool-calling, structured IDK, in-language thinking, RAG integration. |
| v1.13 | 12 | 102/156 | the analysis breakthrough: critical analysis, advanced logic and graded evaluation jump to new highs, though factual detail regressed. |
v1.1 taught us the key lesson — a small model can't memorize every fact, so instead of faking it, v1.11 was retrained to be honest. v1.12 built on that with deeper training and RAG. v1.13 pushes reasoning and analysis skills further, trading some factual detail for gains in logic.
3 · What's New in v1.13
The Analysis & Reasoning Push
New strengths in v1.13:
- ANALYSIS 0/12 → 12/12 — the biggest single gain. Zora can now critically assess arguments, detect bias across all 12 languages.
- LOGIC2 0/12 → 7/12 — advanced multi-step reasoning chains now work in 7 languages.
- GRADED 0/12 → 3/12 — partial progress on essays, assessments and rubrics.
- LOGIC 0/12 → 3/12 — formal logic and syllogisms improve in some languages.
Honest trade-offs:
- DETAIL 10/12 → 2/12 — a significant regression. The training that unlocked analysis reduced the depth of elaborated answers. We report this transparently; it is the clearest weakness of v1.13.
- SEARCH 7/12 → 5/12 — a smaller drop in quick factual lookup.
- FACT 0/12 → 1/12 — marginal improvement in factual recall, still capacity-bound.
- HALLU remains at 10/12 (see the Deep Dive below).
The v1.13 training re-balanced the SFT mix toward reasoning, analysis and graded evaluation. That raised the top end of the scoreboard by +17 points (85 → 102) while costing depth on the DETAIL axis and some speed on SEARCH. It is a deliberate, documented trade — not a free win.
4 · Benchmark (BalkanBench, 13 axes × 12 languages)
🔬 BalkanBench is open — test any model yourself: https://github.com/olivilo/balkanbench Deterministic scoring (script / language / keywords / numbers).
Axis-by-Axis Comparison (v1.12 → v1.13)
| Axis | v1.12 | v1.13 | Δ | What changed |
|---|---|---|---|---|
| FACT | 0/12 | 1/12 | ↑1 | 8B capacity limit; slight improvement, still small |
| HALLU | 10/12 | 10/12 | = | Stable; quality of native-language refusals stays high (see Deep Dive below) |
| DETAIL | 10/12 | 2/12 | ↓8 | Regression — the honest cost of v1.13's reasoning push |
| GRADED | 0/12 | 3/12 | ↑3 | New: essays, assessments, rubrics now sometimes work |
| TEACH | 12/12 | 12/12 | = | Perfect — remains a core strength |
| REASON | 11/12 | 11/12 | = | Strong arithmetic reasoning stays |
| LOGIC | 0/12 | 3/12 | ↑3 | Formal logic / syllogisms improve in some languages |
| LOGIC2 | 0/12 | 7/12 | ↑6 | Multi-step reasoning chains now work in 7 languages |
| ANALYSIS | 0/12 | 12/12 | ↑12 | Biggest gain: critical analysis, bias detection in all 12 languages |
| INSTRUCT | 11/12 | 12/12 | ↑1 | Recovered to perfect instruction following |
| LONGFORM | 12/12 | 12/12 | = | Perfect — remains a core strength |
| SEARCH | 7/12 | 5/12 | ↓2 | Small regression in quick factual lookup |
| TOOLBASE | 12/12 | 12/12 | = | Perfect: answers basics without calling tools |
| TOTAL | 85/156 | 102/156 | +17 | Analysis + logic gains outweigh the DETAIL/SEARCH cost |
Per-Language Scores
| Language | v1.12 | v1.13 | Δ |
|---|---|---|---|
| bg (Bulgarian) | 8/13 | 8/13 | = |
| bs (Bosnian) | 8/13 | 10/13 | +2 |
| cnr (Montenegrin) | 8/13 | 8/13 | = |
| el (Greek) | 9/13 | 9/13 | = |
| hr (Croatian) | 9/13 | 9/13 | = |
| hu (Hungarian) | 8/13 | 7/13 | -1 |
| mk (Macedonian) | 7/13 | 11/13 | +4 |
| ro (Romanian) | 8/13 | 9/13 | +1 |
| sl (Slovenian) | 6/13 | 7/13 | +1 |
| sq (Albanian) | 8/13 | 8/13 | = |
| sr (Serbian) | 7/13 | 8/13 | +1 |
| tr (Turkish) | 7/13 | 8/13 | +1 |
Biggest winner: Macedonian (+4) — the analysis training lifted the previously least-covered language the most. Bosnian (+2) also gained clearly. Hungarian slipped one point.
Charts
| Ranking | Evolution | Axis Matrix |
|---|---|---|
 |
 |
 |
| Delta (v1.12 → v1.13) | What Each Axis Tests |
|---|---|
 |
 |
5 · Deep Dive: Why HALLU Stayed at 10/12
The HALLU score (10/12) stayed flat across v1.12 and v1.13. The IDK behavior itself remains strong — the model says "I don't know" in a structured, native-language way instead of inventing facts. So why didn't it move, and where is the real change?
Why the Score Didn't Move
1. HALLU is already near its ceiling for an 8B model. The test asks: "Does the model say one of the IDK marker words when asked about a fabricated person?" Zora does this correctly for 10 of 12 languages. The last 2 (Macedonian, Slovenian) have the smallest training data — an 8B model simply lacks capacity for these underrepresented languages. Notably, Macedonian improved overall (+4) thanks to the analysis training, but the specific HALLU markers still miss there.
2. IDK training improved QUALITY, not SCORE. BalkanBench's HALLU test is a binary yes/no check for an IDK marker. What actually improved:
| Before (v1.11) | Now (v1.12 → v1.13) |
|---|---|
| Short, sometimes truncated refusals | Full-sentence, structured refusals |
| Sometimes answered in English | Always answers in the question's language |
| No reasoning given | Explains why it can't answer |
| "Ne znam." | "Nemam pouzdanih podataka o 'X'. Ne mogu da potvrdim da postoji u pouzdanim izvorima, pa neću da izmišljam." |
This is a qualitative leap that the binary score cannot capture.
3. The real halluck shift is in ANALYSIS, not HALLU. The v1.13 breakthrough is on the ANALYSIS axis (0/12 → 12/12): Zora now critically evaluates arguments and detects bias across all 12 languages. That is where the reasoning training showed its strongest, most consistent effect.
4. The honest cost: DETAIL regressed. DETAIL fell from 10/12 to 2/12. The same training that unlocked analysis made answers more concise and less elaborated. We report this transparently — v1.13 trades depth of detail for critical-analysis capability. Teams that need long, richly detailed prose should weigh this against the reasoning gains.
What Would Move HALLU to 12/12
| Approach | Expected Impact | Effort |
|---|---|---|
| Larger model (v2 = 27B) | +1-2 languages (mk, sl) | High (new training run) |
| More IDK examples for mk/sl specifically | +0-1 languages | Medium (data generation) |
| RLHF with human feedback on refusal quality | Better quality (not score) | High (human annotation) |
| DPO (Direct Preference Optimization) | +1-2 languages | Medium (preference pairs) |
| Re-balance DETAIL vs ANALYSIS trade | Recover detail at some analysis cost | Medium (data mix tuning) |
Bottom line: The 8B model is near its ceiling for HALLU. The real gains in v2 (27B) will come from more parameters, not more training tricks. v1.13's win is analysis; its documented cost is detail.
6 · 🆕 RAG Feature
Zora integrates with RAG (Retrieval-Augmented Generation) — a system that lets Zora search through a local knowledge base before answering.
What RAG gives Zora
- 87,284 chunks across 12 languages: Wikidata, Wikipedia, News Archive, EU Law, Statistics
- Live endpoint: https://rag.ai.in.rs
- Self-hostable: Clone the pipeline from https://github.com/olivilo/zora-v1.13
How it works
The tool-cascade: Zora first checks its RAG knowledge base (local documents, laws, statistics), then falls back to web search if needed, and finally says "I don't know" if neither helps.
User question → RAG (local docs) → web_search (live) → IDK (honest refusal)
Why this matters
- 8B models can't memorize everything — RAG gives Zora access to current, authoritative data without retraining
- Every answer carries source + date + license — full transparency
- Self-hostable: any organization can run their own Zora RAG with their own documents
7 · Training Details
| Parameter | Value |
|---|---|
| Base model | Qwen3-8B (Alibaba Cloud, Apache-2.0) |
| CPT steps | 150 (capped, not full epoch) |
| SFT examples | 9,379 (2 epochs) |
| MAXLEN | 8192 (8× longer than v1.11) |
| QLoRA | r=16, lora_alpha=16, 4bit |
| Data composition | re-balanced toward reasoning, analysis & graded evaluation |
| Infrastructure | Modal A100-80GB, ~4h total, ~$5-10 |
| Quantizations | Q5_K_M (5.4GB, recommended), Q6_K (6.7GB), Q8_0 (8.7GB) |
8 · Usage
Ollama (recommended):
ollama pull olivilo/zora:v1.13
ollama run olivilo/zora:v1.13
HuggingFace Transformers:
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("sovasoft/zora-v1.13")
model = AutoModelForCausalLM.from_pretrained("sovasoft/zora-v1.13", device_map="auto")
GGUF (llama.cpp / Ollama manual): Download Q5_K_M, Q6_K, or Q8_0 from HuggingFace. Avoid Q4 and below — heavy quantization made the model hallucinate in our tests.
9 · Limitations
- 8B capacity: FACT remains structurally weak — more parameters needed (v2 = 27B)
- DETAIL regression: v1.13's answers are less elaborated than v1.12 — the analysis push cost depth (2/12)
- SEARCH regression: quick factual lookup dropped slightly (5/12)
- Quantization: use Q5_K_M / Q6_K / Q8_0 only. Q4 and below degrade honesty.
- Smaller languages (mk, sl) still have less training data — expect lower consistency
- No real-time knowledge without RAG/web-search — the model's memory has a cutoff date
- Multi-step reasoning is improved but not reliable — always verify critical calculations
10 · Benchmark Transparency & Limitations
BalkanBench is Sovasoft's own benchmark — designed, built, and scored by the same team that built Zora. This means:
- Design bias: The 13 axes (FACT, HALLU, DETAIL, etc.) were chosen to highlight Zora's strengths. A different benchmark design would produce different rankings.
- Scoring bias: The scoring functions in
matrix_ollama.pyare our own. How we define "correct" may favor Zora's training profile. - No frontier comparison: We compare only against open models (7-32B). Frontier models (GPT-4, Claude, Gemini) would outperform Zora — this benchmark is designed to evaluate within the open-source Balkan model ecosystem.
- Selection bias: We include models where Zora competes well. Inclusion criteria are not random.
- Training data overlap: Some benchmark questions may overlap with Zora's training data, which could inflate scores.
What the scores DO show: Zora v1.13 is the strongest open-source model we tested on our benchmark for 12 Balkan languages — 102/156. It outperforms 3-4× larger models on BalkanBench v1.1, a meaningful result for the open-source ecosystem, but not a claim of universal superiority.
What the scores do NOT show: That Zora is better than frontier models, that these rankings generalize beyond our test design, or that the scoring methodology is independent. The transparency here matters most: v1.13 gains analysis at a real, documented cost in factual detail.
10 · What's Next: v2
| v1.13 (now) | v2 (planned) | |
|---|---|---|
| Base | Qwen3-8B | Qwen3.8-27B |
| BalkanBench | 102/156 | Target: 120+/156 |
| DETAIL | 2/12 (regression) | Recover + Target: 8+/12 |
| LOGIC/LOGIC2 | 3/12, 7/12 | Target: 8+/12 |
| HALLU | 10/12 | Target: 12/12 |
| Reasoning | Improved analysis | Full chain-of-thought training |
11 · Acknowledgements
Zora exists because of open source. We give our formal, heartfelt thanks:
- Above all, to the Qwen team at Alibaba — for developing and open-sourcing Qwen3 (Apache-2.0), the foundation model Zora is built upon. Without their generosity, Zora would not exist.
- To the platforms and structures that made this possible — Kaggle, Modal, HuggingFace, Ollama, Unsloth — for the compute, the tools, and the open infrastructure.
- To the open-source community, for the models, code, and knowledge freely shared with everyone.
- To the people of the Balkans — whose languages, voices, stories and perspectives are Zora's very heart.
- To rag.ai.in.rs for the RAG infrastructure and 87,284 chunks of Balkan knowledge.
- And to all that is.
зора — the dawn belongs to everyone.
12 · Citation
@software{zora_v113,
author = {Vignjevic, Oliver},
title = {Zora v1.13: An Open, Honest LLM for the Balkans \& Southeast Europe},
year = {2026},
publisher = {Hugging Face},
url = {https://huggingface.co/sovasoft/zora-v1.13},
license = {Apache-2.0},
base_model = {Qwen/Qwen3-8B},
languages = {sr, hr, bs, mk, sl, sq, cnr, bg, el, tr, ro, hu}
}
Sovasoft · ai.in.rs · one to unite them all
- Downloads last month
- 231