Ostrich 27B 260721-i1 working perfectly!
Ostrich 27B 260721-i1 — the loop / hallucination bug is fixed
Quick follow-up to my last two reports — reran the exact same harness (same header-rewrite prompt, same ~1k / ~12k-char inputs, same sampling: temp 0.6 / top_p 0.95 / min_p 0.05 / top_k 0 / repeat_penalty 1.0) against ostrich-27b-260721-i1, this time testing both thinking-ON and no-think in one pass (we can toggle reasoning per-call now via a /v1/completions prefill, so no more Jinja edits / reloads between runs).
6/6 PASS. No loops, no derailment, no hallucination — including the previously-worst case (no-think on the large ~12k-char input, which used to produce 0% grounding and broken-record repetition to the token cap).
| Variant | Mode | Time | Completion tok | Reasoning | Finish | Verdict |
|---|---|---|---|---|---|---|
| small / mt4096 | think | 15s | 1252 | 4578 chars | stop | ✅ PASS |
| small / mt4096 | no-think | 3s | 242 | 0 | stop | ✅ PASS |
| small / mt16384 | think | 17s | 1409 | 5248 chars | stop | ✅ PASS |
| small / mt16384 | no-think | 3s | 251 | 0 | stop | ✅ PASS |
| large (~12k) / mt16384 | think | 43s | 3737 | 6984 chars | stop | ✅ PASS |
| large (~12k) / mt16384 | no-think | 10s | 1095 | 0 | stop | ✅ PASS |
I spot-checked raw output, not just the auto-classifier:
- Large input, no-think (the worst offender before) — all original content preserved verbatim, every header sensibly rewritten with document-title context injected, clean
finish_reason: stop. - Small input, thinking ON — the reasoning trace is a normal, on-task header-by-header plan that converges straight to the answer. No more "sermon loop."
Whatever changed between 260603 and 260721 fixed the no-think path specifically — that was the one still broken in my last report. This is now solid for our batch RAG-cleanup pipeline in both modes.
Thanks for digging into it!
Side note: how we're toggling thinking now
Worth sharing since it's what let us test both modes in one pass this time. On our LM Studio build, every chat-endpoint switch for Qwen3.x reasoning is silently ignored —chat_template_kwargs: {"enable_thinking": false}, /no_think in the prompt, and reasoning_budget: 0 all still reason regardless of what we pass.
The fix: the legacy /v1/completions endpoint applies no chat template at all ("Prompt template is not applied"), so we build the ChatML prompt ourselves and append the think-scaffold directly instead of asking the template to do it:
- thinking OFF → end the prompt with
<|im_start|>assistant\n<think>\n\n</think>\n\n(a pre-closed, empty think block — model treats reasoning as already done and answers directly) - thinking ON → end the prompt with
<|im_start|>assistant\n<think>\n(open block — model reasons, then closes it and answers)
That's the entire mechanism enable_thinking is supposed to drive internally — we're just injecting it client-side instead of relying on the template/kwarg plumbing, which sidesteps the bug completely and works per-call with no reload. Only real cost: no chat-template extras (tool-call formatting, etc.) on that path, which we don't need for this use case.
Glad that it worked for you. I am also having success and slowly moving my contemplation scripts to use my own model. This will make the model contemplate on familiar texts even better compared to vanilla 3.5 or 3.6.
Merging seems to be healing a lot of overfittings. This gave me an idea: what happens if I merge all the abliterations and agentic fine tunes on HF? A lot of the fine tuners are doing things wrong and ending up being worse than vanilla (in terms of skills that the vanilla is advertised in), but better in a few self bench marks done by the fine tuner probably. So if we merge everything on HF this could smooth out (cancel out) everybody's overfittings..
I am running with llama.cpp as llama-server and giving --reasoning-budget 0 --reasoning off as cli args..
That's very interesting... In my case, I could merge with some of the medical models but they are all so old so I'm not sure how useful the weights are. It might be interesting to investigate special purpose models in certain domains to add value that you want like coding, agents, some technical field, etc.
Keep up the good work which is very appreciated.