ThinCtx-Qwen3.5-4B-small

Quality of Qwen3.5-4B in 4.0/4.7 GB of GPU memory at 128k/540k tokens of context, instead of 12.0/24.5 GB

2.2× smaller weights · GPU memory for the context about 10× smaller than the original's KV cache at 128k tokens and growing about 19× more slowly (1.7 KB instead of 32 KB per token) · the full context in system RAM 2.5× smaller than in the full-precision release · function calling on par with the original (BFCL 89.0 vs 89.5 on the same items) · runs ~540k-token contexts on a 24 GB GPU, where the original does not fit · calibrated and tested on English, Czech and German.

ThinCtx by thinletter. ThinCtx-Qwen3.5-4B-small is the ThinCtx conversion of Qwen3.5-4B that keeps the GPU memory for the context small.

This is the compact variant of ThinCtx-Qwen3.5-4B. The difference: the full-precision release keeps the original's 16-bit weights, while here the weights are quantized too (about 5.4 bits per weight), and the context stored in system RAM is compressed 2.5–3× compared with the 16-bit store. So it needs less GPU and system memory.

Variant	Weights	Context in system RAM
ThinCtx-Qwen3.5-4B	16-bit, as the original	16-bit
ThinCtx-Qwen3.5-4B-small	quantized, about 5.4 bits per weight	compressed, 2.5–3× smaller
ThinCtx-Qwen3.5-4B with --compress_context (thinctx ≥ 1.0.2)	16-bit, as the original	compressed, as in -small

The part of the context kept on the GPU is small and grows slowly with the length of the conversation or document; the full context is kept in ordinary system memory. ThinCtx reads long prompts more slowly than the base model. Chat and tool calling use the Qwen3.5 chat template.

Runtime required. ThinCtx models run on the thinctx runtime (installed in its own environment, below). The runtime is what keeps the context memory small; other inference libraries do not do that with these weights. This model requires thinctx >= 0.3.1. 1.0.1 or newer is recommended (generation with the compact weights 2.6× faster than with 1.0.0 on an RTX 5060 Ti, and it runs on WSL2). With thinctx 1.0 or newer, conversations that start with the same system prompt also share its memory on the GPU and in system RAM (Linux), so each additional conversation costs only a small amount of memory.

Install and run

Platforms. Linux x86_64, or Windows 10/11 through WSL2, with an NVIDIA GPU; macOS is not supported. The default PyTorch 2.11 wheel needs a driver with CUDA 13 support (with an older one, see below). In WSL the Windows NVIDIA driver provides CUDA: do not install a Linux driver inside WSL. Triton compiles a small launcher on the first start and needs gcc: sudo apt install -y build-essential (Debian / Ubuntu) if gcc --version fails.

Linux / WSL:

curl -LsSf https://astral.sh/uv/install.sh | sh && source $HOME/.local/bin/env   # only if uv is missing
uv venv --managed-python --python 3.12 ~/thinctx && source ~/thinctx/bin/activate
uv pip install "thinctx>=1.0.1"
thinctx chat thinletter/ThinCtx-Qwen3.5-4B-small

With an older driver, run uv pip install "torch==2.11.*" --index-url https://download.pytorch.org/whl/cu128 before installing thinctx. Without uv: python3.12 -m venv ~/thinctx (on Debian / Ubuntu it needs the python3.12-venv package, and the python3.12-dev headers for Triton's launcher compile), then pip install "thinctx>=1.0.1".

Windows: in an admin PowerShell wsl --install -d Ubuntu-24.04, restart, open Ubuntu, then the Linux steps. Memory sharing across conversations is Linux-native only (WSL runs without it). WSL keeps its file cache in Windows RAM: add autoMemoryReclaim=dropcache under [experimental] in %UserProfile%\.wslconfig.

An OpenAI-compatible endpoint (add --auth USER:PASS for HTTP basic auth):

thinctx serve thinletter/ThinCtx-Qwen3.5-4B-small --port 8000

import thinctx
engine = thinctx.load("thinletter/ThinCtx-Qwen3.5-4B-small")

The server speaks the OpenAI chat-completions API (/v1/chat/completions, streaming and non-streaming, tools, enable_thinking). Point any OpenAI client at http://host:8000/v1.

Size

	ThinCtx-Qwen3.5-4B	ThinCtx-Qwen3.5-4B-small
Download	8.94 GB	3.63 GB
Bits per weight (transformer layers)	16	5.4

The parameter count Hugging Face shows for this repository counts only the 16-bit tensors in model.safetensors; the quantized weights are stored packed in aux1.safetensors. The model has the same 4.21B parameters as ThinCtx-Qwen3.5-4B.

Results

Measured on these files, on the same items as the other two columns, in separate runs (RTX 4090 / RTX 3090 / RTX 5060 Ti): long context, function calling on all items and the Czech / German rows except the Czech / English text loss with thinctx 1.0.2, the other rows with the thinctx 0.3 runtime. RULER and BABILong: 100 items each. LongBench v1: 16 English tasks with their official metrics. Facts: planted facts asked thousands of tokens later. Tool calls: tool definitions far back in the context, share of calls predicted as in the session. Rules: planted policy rules followed far back in the context (1.00 = as the original). Facts, tool calls and rules are our own tests; their items and scorer are public: thinletter/longctx-agent-bench. BFCL: non-live categories, all items or the first 50 of each. Divergence: KL of the next-token distribution from the original's, lower is better. Czech / English text: held-out parallel passages (FLORES), the increase of the per-token loss (NLL) over the original's. Bits per byte: lower is better. Belebele: multiple-choice reading comprehension. Needle search: RULER-style tasks in Czech and German. RAG: answers from planted documents.

	Qwen3.5-4B	ThinCtx-Qwen3.5-4B	ThinCtx-Qwen3.5-4B-small
Facts recalled at 8k / 32k tokens	0.83 / 0.80	0.86 / 0.80	0.84 / 0.80
Distant tool calls at 8k / 32k tokens	0.94 / 0.99	0.94 / 0.99	0.94 / 0.99
Rules followed at 8k / 32k tokens	1.00 / 1.00	1.00 / 0.90	0.98 / 0.92
Rules kept far back, worst case (1.00 = as the original)	1.00	0.90	0.77
Rules in Czech / German followed at 32k tokens	1.00 / 1.00	0.99 / 1.00	0.91 / 0.99
Czech rules kept far back, worst case	1.00	0.87	0.71
German rules kept far back, worst case	1.00	0.89	0.77
RULER, 4k tokens	98.0	98.7	98.0
RULER, 16k tokens	99.0	98.8	98.8
RULER, 64k tokens	97.0	95.5	98.2
RULER, 128k tokens	98.7	94.6	95.3
BABILong, 4k tokens	70	69	75
BABILong, 16k tokens	63	62	60
BABILong, 64k tokens	57	58	56
BABILong, 128k tokens	57	52	51
LongBench v1, mean of 16 tasks	45.1	45.1	45.2
Czech / English text, prediction loss vs the original	reference	<0.1 % / <0.1 %	+0.3 % / +0.2 %
Czech web text, bits per byte (lower is better)	1.188	1.183	1.189
German web text, bits per byte	0.995	0.987	0.993
Czech EU law, 8k tokens, bits per byte	0.526	0.525	0.527
German EU law, 8k tokens, bits per byte	0.483	0.480	0.486
Czech reading comprehension (Belebele), %	78.3	78.7	78.0
German reading comprehension (Belebele), %	83.0	83.7	82.3
Czech needle search at 64k tokens, %	98.6	100	98.6
German needle search at 64k tokens, %	100	100	97.1
Czech RAG answers at 8k tokens, %	100	100	100
German RAG answers at 8k tokens, %	90	90	90
Function calling (BFCL, non-live), %	86.2	85.1	86.2
Function calling (BFCL non-live, first 50 per category), %	89.5	85.5	89.0
Divergence on plain text, 4k / 16k tokens	reference	0.0019 / 0.0024	0.0072 / 0.0077

Full results for all sizes: https://thinletter.io/thinctx/

Speed

Tokens per second for one conversation on one RTX 3090, with thinctx 1.0.1. The original runs in vLLM 0.31 with CUDA graphs. The 4B-small reads prompts 3.2 to 3.6 times more slowly than the original in vLLM and generates more slowly at 8k and 128k tokens. At 512k tokens the original does not fit on the 24 GB GPU in vLLM (its KV cache alone needs 16 GiB); the 4B-small runs it at 36 tokens per second with 5.2 GB of GPU memory. Speed only: quality beyond 256k tokens was not measured.

	Qwen3.5-4B (vLLM)	ThinCtx-Qwen3.5-4B-small
Generation, 8k tokens	78	50
Generation, 128k tokens	52	44
Generation, 512k tokens	does not fit	36
Prompt reading, 8k tokens	7,734	2,437
Prompt reading, 128k tokens	3,569	1,002
Prompt reading, 512k tokens	does not fit	398

Requirements

Linux x86_64 or WSL2, CPython 3.12, an NVIDIA GPU, PyTorch 2.11, gcc. pip install thinctx pulls torch, transformers, fla-core and safetensors. The default PyTorch 2.11 wheel targets CUDA 13; with an older NVIDIA driver, install the CUDA 12.8 build first: pip install "torch==2.11.*" --index-url https://download.pytorch.org/whl/cu128. causal_conv1d is optional and makes prefill faster (without it a slower pure-PyTorch fallback is used).

Same recipe, other models and languages

• Want the same recipe on another model?
• Want it tuned and calibrated for your language or your data?

Contact us at info@thinletter.io.

License

Released under the PolyForm Small Business License 1.0.0 (LICENSE, full text in LICENSE-PolyForm-Small-Business-1.0.0.txt): free for companies with fewer than 100 people and under USD 1 million revenue (2019 dollars, adjusted for inflation). LICENSE adds free use, for anyone, for personal non-commercial purposes, for non-commercial research and education, and for evaluation and testing (non-production). Production use by larger organisations needs a commercial licence. The weights are a derivative of Qwen3.5-4B (Apache-2.0); Qwen's license (LICENSE-Apache-2.0-Qwen.txt) and a NOTICE with the attribution and the changes are included.

Using ThinCtx at work, or above the free threshold? Write to info@thinletter.io; we will also let you know about new releases.

_{Header painting © Jan Bébar, used with the author's permission.}