S 2.1 in agentic pipelines: restoring reasoning across turns, and a dead end on the looping mechanism
I ran S-2.1 as the model for all 18 roles of a multi-agent coding pipeline across seven multi-hour runs (summary, ledger). Below: what to check if your agent is looping, what real traffic says about #15's tools-and-reasoning finding, two notes on the repos, and a correction to the earlier version of this post.
Two things only @joerowell / @baranowskiadam can answer. Does the hosted OpenRouter endpoint serve the same BF16 weights as this repo? The endpoint says quantization: bf16, but that is what it advertises, not proof of what it loads. And is FP8's stripped generation_config.json intended? Details in the repo notes below.
1. If your agent is looping
Check whether prior reasoning is getting back to the model. The model card says to keep reasoning_content from earlier assistant messages, and the chat template wraps it in <think>...</think> for every past turn. If your client does not send it back, each past turn arrives as an empty <think></think>. Mine did not, for six runs: 18 sessions that looped and never got out on their own. The seventh run was the first where reasoning was actually sent back, and it did not loop once. That is a single run, on a smaller task, with several things changed at the same time, so treat it as the first thing to check rather than as a fix. It is cheap to check and most proxied setups get it wrong. Section 2 has the specific trap and how to keep reasoning alive across turns.
Once it is looping, the loop itself is not as sticky as it looks. I rebuilt the repetition synthetically, as earlier assistant turns, with nothing anywhere telling the model to continue, and then offered it a way out. All 24 runs stopped on their own. In six of the eight runs where no exit was offered, the model reproduced the repeated block exactly once and then stopped, with thousands of tokens still available. Every exit that was offered got taken, 16 out of 16, including one that just changed the subject to an unrelated question. So the urge to copy is real but it fires once, and almost anything else on the table displaces it. Two of the no-exit runs dropped the repetition unprompted and went back to the original task, and both then gave a confident wrong answer, which is worth knowing if you are building something to break loops. Getting out is not the same as recovering.
That flatly contradicts what I saw in a real session, and I cannot explain it. I injected four corrective messages into a live loop and the output stayed character-for-character identical, all four times. Synthetically every exit gets taken; in a real stuck session none of mine did. Same model. If you have a stuck session and a way to inject a message into it, that is the cheapest experiment anyone could run to tell the two apart.
2. Tools and reasoning: #15 is right
#15 found that including a callable tools array kills the reasoning stream, and that tool_choice: "none" brings it back. My own probes reproduced that: 0 of 4 with tools callable, 4 of 4 with tool_choice: "none".
Real traffic does not. Logging every call in a live pipeline, 250 of 330 calls carrying 70 callable tools reasoned anyway. That includes all three calls that opened a session, which reasoned 138, 142 and 105 tokens, and an opening call has no earlier reasoning to send back even in principle. The same pattern held across another 302 calls in a later phase. So "callable tools suppress reasoning" is not true as a blanket rule.
It is also not simply that errands suppress and other prompts do not, because #15's own probe asked "What is 17 * 23?" in every case, which is a calculation rather than an errand, and their callable-tool run still came back with nothing. What squares all of it is that the rate moves around. My own calculation-plus-tools probes ran 4 of 4, then 1 of 4, then 1 of 4, then 3 of 4 in different time windows, and #15's single run sits comfortably inside that spread.
What holds up: I have never seen reasoning suppressed without callable tools present, across eleven runs with no tools array. It happens intermittently rather than always, at a rate that drifts. And it is most consistent on errand-type prompts, meaning "use the run_shell tool to find out X", which came back empty 0 out of 28 times in my probes. Read that 0 of 28 as the extreme end and not the norm, though: my probe errands were trivial one-line lookups, and the closest thing in real traffic, bare tool calls with no text alongside them, came back empty 51% of the time rather than 100%.
What to do about it. Open the session with one turn that sets tool_choice: "none" and asks for reasoning rather than an errand. That gave 4 of 4, and it is something you set on the request rather than a change to how your harness builds the tools array. Then send each turn's reasoning back in reasoning_content on the next turn. Sending the same text back as ordinary content does nothing, so it is the field that matters and not the extra words. Expect it to fade rather than hold: 1601, 746, 309, 70 tokens over four turns. And I have never run the two halves together, so the combination is my inference rather than a result.
If you go through a proxy, one specific trap. LiteLLM writes prior reasoning into assistant_message["thinking_blocks"] in llms/anthropic/experimental_pass_through/adapters/transformation.py (:662-663 on v1.86.2, :594 on v1.94.0), and llms/openrouter/chat/transformation.py never reads it back out, so the field the template is looking for never gets filled. I have a fix open on my fork (jwbron/litellm#11), not upstream yet. And on the hosted route, the endpoint accepts only [reasoning, include_reasoning, tools, tool_choice, temperature, stop, max_tokens]. Everything else, DRY included, is dropped without telling you, so temperature is your only sampler knob there.
3. Two notes on what the 07-22 commit did
If you run BF16, that fix was not yours. Diffing 17cacdc6 ("spinquantless FP8 ... fixes agentic looping"), config.json changes in one key only, quantization_config. max_position_embeddings (262144) and both rope_parameters blocks are identical before and after. It was a requantization that removed a SpinQuant rotation, and that rotation is something FP8 quantization introduced in the first place. BF16 never had it. So BF16 reports continuing after 07-22 are not evidence that fix failed; it fixed something BF16 does not have.
If you run FP8, check your output limit. That commit re-uploaded the whole repo and overwrote generation_config.json with a much shorter file. Seven minutes later fc72eb90 put the chat template back; nobody put the generation config back, and fc72eb90 is still FP8 HEAD. It picked up max_new_tokens: 8192, which vLLM reads and turns into the default output limit for any request that does not set its own. BF16, NVFP4 and INT4 have no such key. It also lost reasoning_parser, tool_call_parser, default_chat_template_kwargs and speculative_config, which matters less than it looks, since vLLM only reads six keys out of that file (get_diff_sampling_param in vllm/config/model.py) and those four are not among them; they have always had to be command-line flags. But FP8 is the only one of the four repos that lost them and the only one that gained a cap, which reads as a re-upload accident. To be clear, none of this explains the reports that continued in #1, since those users all set the parsers on the command line anyway.
Correction to the earlier version of this post
That version argued the loop was kept alive by the 36 of 48 layers that only see the last 512 tokens, on the grounds that mid-loop those layers contain nothing but copies of the same cycle. I no longer think that is right. A loop I measured later repeated on a 1054-token cycle, twice the window, so those layers never held even one full copy of it. And the synthetic test in section 1 found the copying fires once rather than sticking. The copying reproduces; the part that needed explaining, why a real session keeps at it for an hour and a half, does not. The experiments, including the scoring bugs that made my first attempt worthless, are on the ledger.
holy waflle
i read this whole thing for some reason and i still have no idea what point your trying to make other than please check the fp8 generation_config
Cleaned this up a bit. I had some theories on why the model was looping which I later falsified. It turns out the issues I was seeing in large part can be explained by the reasoning issue described above.