jsh-itsolution/ONDO-27B-LoRA

ONDO-27B (LoRA adapter)

Developed by ITSOLUTION Co., Ltd. (Seoul, South Korea) · https://ondo.sh

ONDO-27B is Qwen3.8-27B fine-tuned to say it does not know instead of guessing on short factual questions it cannot answer, while a KL penalty keeps its next-token distributions close to the base model's. It was trained on 244 of the base model's own answers (its abstentions on questions it got wrong and its correct answers on questions it knew), so the training text is the base model's own reasoning and wording. On the public AA-Omniscience questions its accuracy stayed the same while the hallucination rate fell from 0.239 to 0.170 (one run, see Evaluation).

Files and identity

Item	Value
Base (BF16, trained on)	Qwen/Qwen3.8-27B, revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
FP8 source (merged into)	Qwen/Qwen3.8-27B-FP8, revision 017b9c7af6b5689d5dd426a76e0bc077eb5ca20a
Merged FP8 checkpoint	jsh-itsolution/ONDO-27B-FP8, about 30.9 GB, same layout as the FP8 source including the MTP layers
LoRA adapter	This repository: adapter_model.safetensors SHA-256 72917e83dd41db8fbd18aac3ea69c0c63b10cbcf78ecd7974eb009426c01c8e6, adapter_config.json SHA-256 17141364eaec9e32bede694e13c3dbb045c7708901ba7226f94c2f72ff9df4d6
Training rows	244 of a fixed set of 245 (the trainer's sampler drops the last incomplete batch of four)

Training

• QLoRA on the BF16 base: NF4 4-bit with double quantization, compute dtype bfloat16; LoRA rank 16, alpha 32, dropout 0.05 on all linear layers of the language model (vision tower and aligner frozen).
• Loss per supervised position: cross-entropy on the base model's own answer plus the forward KL divergence KL(base ‖ model) over the full vocabulary at temperature 1.0, weight 1.0 each. The teacher is the same network with the adapter disabled, so it stays the base model throughout. The loss is computed 256 positions at a time, so the full logits are never held at once; on a small model it matches ms-swift 4.5.3's own GKD loss (relative difference 0.0 in the loss, 5.9e-7 in the gradients).
• One epoch, 61 updates: batch 1 × gradient accumulation 4, learning rate 1e-4 constant after 5 warmup steps, max gradient norm 0.3, max length 3,072 tokens, thinking enabled in the chat template, seed 20261029. The KL to the base was 0 for the first two updates and 0.004–0.007 per position over the last five.
• The Gated DeltaNet layers ran on the PyTorch reference path, not flash-linear-attention 0.5.2, whose backward pass gives wrong gradients in early layers under this setup (fla issue #1598, reproduced here).
• Merge: the adapter was added to the FP8 checkpoint in fp32 and re-quantized with an exact-midpoint tie rule. All 496 target modules changed; every other tensor equals the FP8 source and the MTP shard file is unchanged.

Training data

• Questions: PopQA (MIT), EntityQuestions (MIT) and Mintaka (CC-BY-4.0; Amazon, "Mintaka: A Complex, Natural, and Multilingual Dataset for End-to-End Question Answering", COLING 2022).
• Answers: the base model's own samples under the AA-Omniscience public answer prompt (the system message); no other model wrote any training text.
• Questions identical to, or sharing an English 8-gram with, AA-Omniscience's public 300 or SimpleQA Verified were removed before sampling. TriviaQA was left out because its distribution states no terms.
• No evaluation set named below was used for training, development or threshold setting.

Evaluation

All numbers are our own measurements on our own GPUs, not Artificial Analysis results.

Benchmark	This model	Base (same harness)	Notes
AA-Omniscience v1.1, public 300	accuracy 0.137, hallucination rate 0.170, index contribution 5.517	accuracy 0.135, hallucination rate 0.239, 5.155 (two runs)	Difference +0.362 points (+0.070 if every cut answer counts against this model; s.d. 0.206). This model ran on a test engine, the base on the serving path, so the path effect is not separated
HLE-Diamond, fixed 150 items (screen)	19/150 = 0.127 (one run)	38/300 = 0.127 (two runs, 65,536-token cap, a cut answer counts as wrong)	Pre-registered screen: passes with 18 or more of 150 correct, i.e. no drop larger than 0.3 index points
Korean prompts that did not ask for Chinese or Japanese script (473): answers containing Han characters or kana	87	73	The rise is not significant under the pre-registered one-sided test (p 0.097; 57 answers in this model only, 43 in the base only). This does not show the two are equal

Recommended settings

Use the base model card's settings for thinking mode (temperature 1.0, top_p 0.95, top_k 20, min_p 0, presence_penalty 0). The HLE-Diamond screen used them with reasoning effort xhigh.

Limits

• Trained on 244 rows of English short factual questions; the change it makes outside that kind of question is not measured beyond the rows above.
• The evaluations are small public samples under an output cap of 65,536 tokens, not the published benchmark conditions.
• Abstaining more lowers the answer rate (not attempted 63.8% → 70.0% on the public 300); accuracy did not fall there.

License

Apache-2.0, as the base model. Mintaka questions are CC-BY-4.0 and are credited above.