trymirai/Qwen3.8-27B-S-experimental

Qwen3.8-27B, Mirai S, experimental

Qwen3.8-27B quantized with the Mirai S codec: a QTIP trellis at 2.4 bits per weight, 8.45 GB on disk, tuned against the teacher's KL.

This is a work in progress and the numbers here are not final. Work on both the lalamo and uzu sides is unfinished. Treat anything measured with it as provisional.

Neither runtime can load this package from a released version yet, so each section below names the branch you need. The two runtimes produce identical output: on the same prompt at temperature 0 they agree token for token.

Apple silicon — uzu

The S trellis kernels are not upstream. You need ryan/qtip-s-agent, at commit a0850ffa or later (earlier commits reject this package's config):

git clone https://github.com/trymirai/uzu.git && cd uzu && git checkout ryan/qtip-s-agent

Point the serve script at wherever you downloaded this repo:

scripts/serve-mirai-s.sh /path/to/Qwen3.8-27B-S-experimental

That builds the CLI on the first run, then serves an OpenAI-compatible endpoint on port 8000:

curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"ours","messages":[{"role":"user","content":"Capital of Australia? One word."}],"max_tokens":8}'

Metal kernels, so Apple silicon only. Roughly 340 tokens/s prefill and 16 tokens/s decode at 7k context on an M5 Pro.

The script serves with the thinking trace off, which is what agent work wants. Turn it on per request with "enable_thinking": true, or server-wide with UZU_SERVER_THINKING=1. Two things to know:

• reasoning_effort on its own returns 400 under the script's default, because the server default reads as an explicit enable_thinking: false. Send both, or start the server without UZU_SERVER_THINKING=0.
• uzu's levels are low / medium / high, not the package's. They barely change trace length in practice.
• Streaming drops the reasoning trace: reasoning_content is absent from deltas, though the tokens are still generated and still cost you the latency. Use non-streaming when you want to see traces.

NVIDIA — lalamo

lalamo runs this package on CUDA. The weights stay compressed on the GPU: 8.45 GB after loading, 9.25 GB at peak while generating, and it runs under a hard 10 GB memory cap. Any NVIDIA card with 12 GB or more works.

It is a research toolkit, not optimised for inference — expect roughly 3 minutes for 40 tokens on a B200. Use it to check quality, not speed.

git clone -b ry2009/s-checkpoint-lalamo https://github.com/trymirai/lalamo.git && cd lalamo

uv sync --extra cuda --extra server

XLA_PYTHON_CLIENT_PREALLOCATE=false uv run lalamo server --port 8293 --batch-size 1

--batch-size 1 is required. Without it the server falls back to VRAM-based batch estimation, which corrupts donated buffers on CUDA and fails every request with Array has been deleted.

XLA_PYTHON_CLIENT_PREALLOCATE=false stops JAX from reserving 75% of the card up front. On a 12 GB card that reservation is 9 GB, just under the 9.25 GB peak.

Talk to it with the OpenAI client. model is the path to this repo on disk:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8293/v1", api_key="none")

reply = client.chat.completions.create(
    model="/path/to/Qwen3.8-27B-S-experimental",
    messages=[{"role": "user", "content": "Capital of Australia? One word."}],
    max_completion_tokens=40,
)

message = reply.choices[0].message
print("thinking:", message.reasoning_content)
print("answer:  ", message.content)

uv run --with openai python try_it.py

Long outputs need more memory for the attention cache: about 64 KB per token of context (16 attention layers, 4 KV heads of 256), so a 32k-token reasoning run adds roughly 2 GB. That figure is computed from the architecture, not measured.

Streaming works but is not incremental — the finished reply arrives as one chunk. Tool calling is not supported and returns a 400 rather than silently ignoring the tools.

Reasoning effort

Through lalamo, reasoning_effort takes xhigh, medium, low or no_reasoning.

effort	system prompt	thinking
medium (default)	none	on
xhigh	"think carefully, validate key assumptions…"	on
low	"keep your thinking brief…"	on
no_reasoning	none	off

The default is medium deliberately: it sends no system prompt, which is what uzu does, so the two runtimes stay comparable. xhigh produces noticeably terse, telegraphic traces — that is the prompt's effect on the model, not damage from quantization; uzu reproduces the same text verbatim when given the same system prompt.

What it is

Base	Qwen3.8-27B, bf16
Rate	2.4 bits per weight
On disk	8.45 GB
KL to the teacher	0.062
Layout	QTIP trellis leaves, per-row scales and QAT gains folded at load
Incoherence	full-dimension RHT applied to activations at runtime; weights stored rotated

encoding.json ships with the package so uzu picks up the chat format without extra setup. physical_package_report.json and provenance.json carry the build lineage; loader_validation.json is the per-leaf check against the dense checkpoint it was packed from.

Speculator

The DFlash Weaver drafter ships in speculator/, which is where uzu looks for it (<model>/speculator with its own config.json and model.safetensors). Download the repo and it is already in place. It changes decode speed only, not what the model produces; delete the directory to run without it. lalamo does not use it.

Known issues

• lalamo: content includes the raw <|im_end|> stop token on every response.
• lalamo: with no_reasoning, the answer arrives in chain_of_thought and response comes back empty — the parser splits on </think>, which the prompt has already closed.
• uzu: streaming never emits reasoning_content, so thinking traces are invisible to streaming clients.
• Both: a tight max_tokens with thinking on can spend the whole budget inside the trace and return an empty answer. Give reasoning requests at least 4000 tokens.