poolside

/

Laguna-S-2.1

like 987

Follow

Poolside 1.49k

Six strings are single tokens in Laguna's vocabulary:

<think>   </think>   <assistant>   </assistant>   <tool_call>   </tool_call>

When one of them appears as data — in source code the model is asked to read
or write — the model cannot reproduce it. It substitutes something else, and in
the worst case it emits </tool_call> mid-payload, which terminates its own tool
call and produces unparseable JSON.

This reproduces on your hosted API, so it is not a serving or quantisation issue
on our side.

Minimal reproduction

curl -s $ENDPOINT/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "'"$MODEL"'",
  "messages": [{"role": "user", "content":
    "Call the bash tool with exactly this command: grep -c \"<think>\" /tmp/a.cs"}],
  "tools": [{"type": "function", "function": {"name": "bash",
    "description": "Run a shell command",
    "parameters": {"type": "object",
      "properties": {"command": {"type": "string"}},
      "required": ["command"]}}}],
  "tool_choice": "auto", "max_tokens": 2000, "temperature": 0}'

Expected grep -c "<think>" /tmp/a.cs, delivered grep -c " " /tmp/a.cs.
The marker is replaced by a single space.

Same result on three endpoints

Probe 2 is byte-identically wrong on all three. Local runs used
Laguna-S-2.1-INT4 under vLLM 0.25.1 with the flags from the model card;
the hosted runs used poolside/laguna-s-2.1 and poolside/laguna-s-2.1:free.

The most informative failure

Asked to produce this line as a tool argument:

int closeIdx = content.IndexOf("</think>", StringComparison.Ordinal);

the hosted API returns, verbatim:

{"filePath": "/tmp/x.cs", "oldString": "            int closeIdx = content.IndexOf(\"</tool_call>

It reached for </think>, wrote </tool_call> instead, and the generation stops
there — the marker terminated the tool call from inside the JSON string.

The confusion also shows up in the model's own reasoning. Asked to echo the
literal X<think>Y, it wrote:

Okay, the user wants me to return exactly the string "XY" and
nothing else.

It read one marker and reported a different one back.

Token-level detail

<think> is a single vocabulary entry, so a user-typed occurrence in a prompt
becomes the same token the chat template uses to open a reasoning block:

'X<think>Y'                    -> [2, 125, 18, 126]
'grep -c "<think>" /tmp/a.cs'  -> [2, 21565, 419, 136, 444, 18, 71, 778, 6684, 7935, 11905]

And in the model's output the token is simply absent — comparing the correct
sequence against what was generated:

correct : ... 136(c)  444(") 18(<think>) 71(")  778(/tmp) ...
measured: ... 136(c)  444(") 444(")            778(/tmp) ...

Zero of three runs at temperature 0 contained token 18. Same with
skip_special_tokens=false and with enable_thinking=false.

Our reading: the marker occurs in training essentially only in structural
position, never as payload inside a string, so there is little signal for
"produce this as content". Whether that is the right explanation is your call —
the measurements above are what we can show.

Why this matters in practice

Any coding agent editing source that contains these strings fails. That mostly
means code which itself handles chat protocols — parsers, chat templates,
harness code. In our case an agent silently made no change at all while
reporting success; across 57 dispatches on such a codebase, 9 produced no edit.

This may also explain #29 ("Laguna S 2.1 ends with no response"), where the
symptom is the same empty response in OpenCode.

Things that do not help

Each measured individually: enable_thinking:false per request (the model then
truncates after one character), reasoning_effort:"none", and
skip_special_tokens:false (the token is never produced in the first place).

Workaround we are using

A harness-level transport encoding: rewrite the six strings to a sentinel in tool
results before the model sees them, and decode them in tool arguments before
anything executes. The model then never encounters the markers and never has to
produce them. Measured on a refactor that previously failed deterministically:
0 of 2 correct without, 4 of 4 with.

That is a workaround, not a fix — it only helps inside our own harness.

Not verified

• We did not test all six markers individually. Token 18 (<think>) is measured
  directly; that 26 (</tool_call>) is affected follows from the model emitting
  it in the wrong place, not from an isolated test.
• We have no view of your training data, so the explanation above is inference
  from behaviour.