ThinCtx-Qwen3.5-4B

ThinCtx by thinletter. ThinCtx-Qwen3.5-4B is the ThinCtx conversion of Qwen3.5-4B that keeps the GPU memory for the context small: it grows about 19 times more slowly than the base model's cache (1.0 GB instead of 16.5 GB at 540k tokens), with accuracy on par with the base model (on average 81.9 % against the base model's 82.5 % on BABILong and RULER at 4k, 16k and 64k tokens and BFCL non-live).

Variants. This release keeps the original's 16-bit weights and stores the context in system RAM at 16 bit. ThinCtx-Qwen3.5-4B-small quantizes the weights too (about 5.4 bits per weight) and compresses the stored context 2.5–3×, so it needs less GPU and system memory. With thinctx 1.0.2 or newer this release also runs with the compressed context store (--compress_context, below).

Variant	Weights	Context in system RAM
ThinCtx-Qwen3.5-4B	16-bit, as the original	16-bit
ThinCtx-Qwen3.5-4B-small	quantized, about 5.4 bits per weight	compressed, 2.5–3× smaller
ThinCtx-Qwen3.5-4B with --compress_context (thinctx ≥ 1.0.2)	16-bit, as the original	compressed, as in -small

The part of the context kept on the GPU is small and grows slowly with the length of the conversation or document; the full context is kept in ordinary system memory. ThinCtx reads long prompts more slowly than the base model (see Speed below). Chat and tool calling use the Qwen3.5 chat template.

Runtime required. ThinCtx models run on the thinctx runtime (installed in its own environment, below). The runtime is what keeps the context memory small; other inference libraries do not do that with these weights. This model requires thinctx >= 0.2.0 (1.0.2 or newer for --compress_context). 1.0.1 or newer is recommended (shared memory across conversations, and it runs on WSL2).

Install and run

Platforms. Linux x86_64, or Windows 10/11 through WSL2, with an NVIDIA GPU; macOS is not supported. The default PyTorch 2.11 wheel needs a driver with CUDA 13 support (with an older one, see below). In WSL the Windows NVIDIA driver provides CUDA: do not install a Linux driver inside WSL. Triton compiles a small launcher on the first start and needs gcc: sudo apt install -y build-essential (Debian / Ubuntu) if gcc --version fails.

Linux / WSL:

curl -LsSf https://astral.sh/uv/install.sh | sh && source $HOME/.local/bin/env   # only if uv is missing
uv venv --managed-python --python 3.12 ~/thinctx && source ~/thinctx/bin/activate
uv pip install "thinctx>=1.0.2"
thinctx chat thinletter/ThinCtx-Qwen3.5-4B

With an older driver, run uv pip install "torch==2.11.*" --index-url https://download.pytorch.org/whl/cu128 before installing thinctx. Without uv: python3.12 -m venv ~/thinctx (on Debian / Ubuntu it needs the python3.12-venv package, and the python3.12-dev headers for Triton's launcher compile), then pip install "thinctx>=1.0.2".

Windows: in an admin PowerShell wsl --install -d Ubuntu-24.04, restart, open Ubuntu, then the Linux steps. Memory sharing across conversations is Linux-native only (WSL runs without it). WSL keeps its file cache in Windows RAM: add autoMemoryReclaim=dropcache under [experimental] in %UserProfile%\.wslconfig.

An OpenAI-compatible endpoint (add --auth USER:PASS for HTTP basic auth):

thinctx serve thinletter/ThinCtx-Qwen3.5-4B --port 8000

import thinctx
engine = thinctx.load("thinletter/ThinCtx-Qwen3.5-4B")

The server speaks the OpenAI chat-completions API (/v1/chat/completions, streaming and non-streaming, tools, enable_thinking). Point any OpenAI client at http://host:8000/v1.

Compressed context

With thinctx 1.0.2 or newer this release can also run with the compressed context store of ThinCtx-Qwen3.5-4B-small: the weights stay 16-bit, and the context stored in system RAM becomes about 2.5× smaller. The runtime fetches only the -small's small store files (a few MB, not its weights) and checks that they belong to this conversion.

thinctx chat thinletter/ThinCtx-Qwen3.5-4B --compress_context
thinctx serve thinletter/ThinCtx-Qwen3.5-4B --compress_context --port 8000

engine = thinctx.load("thinletter/ThinCtx-Qwen3.5-4B", compress_context=True)

At 540k tokens the context then needs 7.8 GB of system RAM instead of 19.8 GB (chart above), and generation is about 5 % slower than with the 16-bit context. Its results are in the --compress_context column of the tables under Results.

Results

Every number compares ThinCtx with the unchanged Qwen3.5-4B on the same items.

GPU memory for the context

Per conversation, model weights excluded, in GB, measured with thinctx 1.0.2 on one RTX 3090 at 131,072 (128k) and 541,584 (540k) tokens. The base model keeps its whole context cache on the GPU; ThinCtx keeps a small part on the GPU, growing about 19 times more slowly, and the full context in system memory (RAM). GPU: what the runtime holds for the context (during the prefill of a long input the peak is higher); system memory: the stored context and its index. With --compress_context the GPU part is about the same.

Test	Qwen3.5-4B	ThinCtx	--compress_context
GPU, 128k tokens of context	4.0	0.38	0.38
GPU, 540k tokens of context	16.5	1.03	1.03
System memory, 128k tokens of context	–	4.79	1.88
System memory, 540k tokens of context	–	19.77	7.78

Speed

Tokens per second for one conversation on one RTX 3090, bf16, with thinctx 0.3.1 (steady decode rate; not re-measured with 1.0.1). The original runs in vLLM 0.31 with CUDA graphs, batch 1. ThinCtx reads prompts 2.4 to 3.2 times more slowly than the original in vLLM and generates more slowly at 8k and 128k tokens. At 512k tokens the original does not fit on the 24 GB GPU in vLLM (its KV cache alone needs 16 GiB); ThinCtx runs it at 33 tokens per second with 10.8 GB of GPU memory. Speed only: quality beyond 256k tokens was not measured.

	Qwen3.5-4B (vLLM)	ThinCtx-Qwen3.5-4B	--compress_context
Generation, 8k tokens	78	43	41
Generation, 128k tokens	52	40	37
Generation, 512k tokens	does not fit	33	31
Prompt reading, 8k tokens	7,734	3,259	2,942
Prompt reading, 128k tokens	3,569	1,112	1,061
Prompt reading, 512k tokens	does not fit	426	403

--compress_context: thinctx 1.0.2; –: not measured.

Long context

BABILong: reasoning over facts hidden in long text. RULER: needle search, tracking and aggregation. LongBench v1: 16 English tasks with their official metrics. Scores in %.

Test	Qwen3.5-4B	ThinCtx	--compress_context
BABILong, 4k tokens	70	69	71
BABILong, 16k tokens	63	62	62
BABILong, 64k tokens	57	58	54
BABILong, 128k tokens	57	52	55
RULER, 4k tokens	98.0	98.7	97.7
RULER, 16k tokens	99.0	98.8	98.8
RULER, 64k tokens	97.0	95.5	96.3
RULER, 128k tokens	98.7	94.6	97.6
LongBench v1, mean of 16 tasks	45.1	45.1	45.0

--compress_context: the 128k rows and LongBench measured on the released files with thinctx 1.0.2; 4k to 64k tokens on the same items in the compressed store's validation run (its own 16-bit reference differs from the ThinCtx column by up to 2 points on the 0.8B and the 2B).

Tools and agent behaviour

BFCL: the Berkeley Function Calling Leaderboard, non-live categories (all items, or the first 50 of each category). Facts: planted facts asked thousands of tokens later, in sessions of 8k and 32k tokens. Tool calls: tool definitions far back in the context, share of calls predicted as in the session. Rules: planted policy rules for an agent's commands, their effect kept far back in the context (1.00 = as the base model). Divergence: KL of the next-token distribution from the base model's, lower is better. Facts, tool calls and rules are our own tests; their items and scorer are public: thinletter/longctx-agent-bench.

Test	Qwen3.5-4B	ThinCtx	--compress_context
Function calling (BFCL, non-live), %	86.2	85.1	85.1
Function calling (BFCL non-live, first 50 per category), %	89.5	85.5	–
Facts recalled at 8k / 32k tokens	0.83 / 0.80	0.86 / 0.80	0.84 / 0.80
Distant tool calls at 8k / 32k tokens	0.94 / 0.99	0.94 / 0.99	0.94 / 0.98
Rules followed at 8k / 32k tokens	1.00 / 1.00	1.00 / 0.90	1.00 / 0.89
Rules kept far back, worst case (1.00 = as the original)	1.00	0.90	0.86
Distant tool calls, divergence from the original (lower is better)	reference	0.0022	–
Plain text, divergence from the original, 4k / 16k tokens (lower is better)	reference	0.0019 / 0.0024	0.0021 / 0.0028

--compress_context: BFCL on all items measured on the released files with thinctx 1.0.2; the other values on the same items in the compressed store's validation run (its own 16-bit reference differs from the ThinCtx column by up to 2 points on the 0.8B and the 2B); –: not measured.

Czech and German

Czech / English text: held-out parallel passages (FLORES), the increase of the per-token loss (NLL) over the base model's. Bits per byte: lower is better. Belebele: multiple-choice reading comprehension. Needle search: RULER-style tasks in Czech and German. RAG: answers from planted documents. Rules: the rules tests with the rule written in Czech or German; items and scorer: thinletter/longctx-agent-bench.

Test	Qwen3.5-4B	ThinCtx	--compress_context
Czech / English text, prediction loss vs the original	reference	<0.1 % / <0.1 %	–
Czech web text, bits per byte (lower is better)	1.188	1.183	1.182
German web text, bits per byte	0.995	0.987	0.986
Czech EU law, 8k tokens, bits per byte	0.526	0.525	0.525
German EU law, 8k tokens, bits per byte	0.483	0.480	0.480
Czech reading comprehension (Belebele), %	78.3	78.7	78.3
German reading comprehension (Belebele), %	83.0	83.7	82.7
Czech needle search at 64k tokens, %	98.6	100	98.6
German needle search at 64k tokens, %	100	100	98.6
Rules in Czech / German followed at 32k tokens	1.00 / 1.00	0.99 / 1.00	1.00 / 1.00
Czech rules kept far back, worst case	1.00	0.87	0.86
German rules kept far back, worst case	1.00	0.89	0.86
Czech RAG answers at 8k tokens, %	100	100	100
German RAG answers at 8k tokens, %	90	90	90

--compress_context: bits per byte, Belebele, needle search and RAG measured on the released files with thinctx 1.0.2; the rules on the same items in the compressed store's validation run (the rules kept far back in the same run as the ThinCtx column); –: not measured.

Full results for all sizes: https://thinletter.io/thinctx/

Requirements

Linux x86_64 or WSL2, CPython 3.12, an NVIDIA GPU, PyTorch 2.11, gcc. pip install thinctx pulls torch, transformers, fla-core and safetensors. The default PyTorch 2.11 wheel targets CUDA 13; with an older NVIDIA driver, install the CUDA 12.8 build first: pip install "torch==2.11.*" --index-url https://download.pytorch.org/whl/cu128. causal_conv1d is optional and makes prefill faster (without it a slower pure-PyTorch fallback is used).

License

Released under the PolyForm Small Business License 1.0.0 (LICENSE, full text in LICENSE-PolyForm-Small-Business-1.0.0.txt): free for companies with fewer than 100 people and under USD 1 million revenue (2019 dollars, adjusted for inflation). LICENSE adds free use, for anyone, for personal non-commercial purposes, for non-commercial research and education, and for evaluation and testing (non-production). Production use by larger organisations needs a commercial licence. The weights are a derivative of Qwen3.5-4B (Apache-2.0); Qwen's license (LICENSE-Apache-2.0-Qwen.txt) and a NOTICE with the attribution and the changes are included.

Using ThinCtx at work, or above the free threshold? Write to info@thinletter.io; we will also let you know about new releases.

_{Header painting © Jan Bébar, used with the author's permission.}