litert-community/clef-flash-LiteRT

Clef-Flash for LiteRT

Cloudflare/clef-flash is a decision model by Cloudflare, post-trained from Qwen/Qwen3.5-9B. It reads a state (text, a JSON value, images or video) and typed questions about it: yes/no (noul), a pick from named options (choice) and an ordered rating (score). For each question it returns the probability of every option. A joint head reads the final hidden states of the whole request and scores every option of every question at once. It never generates text. Requests and responses follow the provider's systemone(), a SystemOne-compatible request shape: a state and questions in, typed answers and a token count out. What it is for and how it scores: the source model card and the blog post.

This repository is a community conversion of the model to LiteRT for desktop runtimes, made with litert-torch 0.9.4 for the CompiledModel API of ai-edge-litert 2.2.0. Five row graphs, decoder graphs for requests of up to 256, 512, 1,024, 2,048 and 4,096 tokens, take the embeddings of a request's tokens and its rotary tables, and return the hidden states after the final norm at every position. Two vision towers turn a picture, read at a 256 px or a 448 px square tile, into 64 or 196 rows of picture embeddings. A Python host builds the request's tokens, writes the embeddings, computes the positions, picks the graph and runs the joint head on the hidden states. The repository also holds the tokenizer, the tables the host reads, the joint head's weights, test fixtures and the conversion scripts.

On an Apple M4 Max with Metal at float32 precision, one decision request of 236 tokens takes a median 360.7 ms and one of 1,007 tokens 1,346.9 ms, including the joint head. One call of the 1,024-token row graph, a 1,024-token prefill, takes 1,308.5 ms. The model decodes nothing: each request is one prefill (2026-10-21).

The reference for every check is the provider's own code in float32 on the CPU. End to end through the host on the Apple M4 Max, on Metal at float32 precision, every request gave the reference's most likely option on every test question: 545 of 545 questions that are not near ties, and all 5 near ties, on 243 runs. Its probabilities differed from the reference by at most 1.08e-5 on text and by at most 1.65e-3 with a picture, where the vision tower's float16 weights add the difference.

This model was measured on a desktop only, one Mac. GPUs on Linux or Windows were not measured. It was not run on a phone and is not expected to fit one: one row graph holds about 13.8 GB of float16 weights, more than the memory of a 12 GB phone.

The example request of this repository, one drawn picture and three questions (examples/request.json, picture fixtures/images/clock_3oclock_256x240.png):

{"model": "clef-flash",
 "state": "The attached image shows a wall clock.",
 "questions": {
  "hour": {"type": "choice", "instructions": "What hour does the clock show?",
           "criteria": {"12": "twelve", "3": "three", "6": "six", "9": "nine"}},
  "minute_hand_up": {"type": "noul", "instructions": "The minute hand points straight up."},
  "confidence": {"type": "score", "instructions": "How clearly can the time be read?",
                 "criteria": ["unclear", "fairly clear", "very clear"]}}}

Read at the 256 px tile, it is 405 tokens and runs on the 512-token row graph. On Metal at float32, the host printed this response, the same as the provider's code in float32 on the CPU, field for field:

{"model": "clef-flash",
 "answers": {
  "hour": {"type": "choice", "choice": "3", "confidence": 0.966,
           "probabilities": {"12": 0.0159, "3": 0.966, "6": 0.0078, "9": 0.0102}},
  "minute_hand_up": {"type": "noul", "noul": 0.9484},
  "confidence": {"type": "score", "score": 1.8549, "confidence": 0.8745,
                 "legend": {"0": "unclear", "1": "fairly clear", "2": "very clear"},
                 "probabilities": {"0": 0.0196, "1": 0.1059, "2": 0.8745}}},
 "usage": {"input_tokens": 405, "output_tokens": 0}}

Files

File	Bytes	Role
clef-flash_rowprefill_embeds_L256_fp16fc.tflite	13,847,841,008	Row graph for requests of up to 256 tokens.
clef-flash_rowprefill_embeds_L512_fp16fc.tflite	13,850,105,024	Row graph for requests of up to 512 tokens.
clef-flash_rowprefill_embeds_L1024_fp16fc.tflite	13,856,205,536	Row graph for requests of up to 1,024 tokens.
clef-flash_rowprefill_embeds_L2048_fp16fc.tflite	13,874,698,016	Row graph for requests of up to 2,048 tokens.
clef-flash_rowprefill_embeds_L4096_fp16fc.tflite	13,936,871,792	Row graph for requests of up to 4,096 tokens.
clef-flash_vision_tower_g256_fp16fc.tflite	907,587,008	Vision tower and merger for a 256 px tile (64 picture tokens).
clef-flash_vision_tower_g448_fp16fc.tflite	907,891,136	Vision tower and merger for a 448 px tile (196 picture tokens).
tables/embed_tokens.safetensors	2,034,237,544	The token embedding in bfloat16, 248,320 × 4,096. The host writes its rows into the row graphs.
tables/lm_head.safetensors	2,034,237,536	The output embedding in bfloat16, 248,320 × 4,096. The joint head reads the rows of each option's tokens.
tables/vision_pos_g256.safetensors, tables/vision_pos_g448.safetensors	1,179,728 / 3,612,752	The vision towers' position rows in float32, resampled to each tile.
head/joint_head.safetensors, head/joint_head_config.json	243,538,016 / 119	The provider's joint head, unchanged.
tokenizer/tokenizer.json	19,989,325	The source repository's tokenizer.json, unchanged.
contract.json		The host contract as data: token ids, graph inputs and outputs, lengths, the picture path, the positions, the head and each file's checks.
host/		Python host clef_litert (requests, positions, pictures, tables, joint head, response, LiteRT runtime) and requirements-host.txt.
examples/		run_example.py, its README, the example request and run_example.expected.json, the provider's answers to the four requests of --check.
fixtures/		Test requests (219 records), their pictures, the provider's reference probabilities for 243 runs, and a README with sources and licenses.
conversion/		Conversion, check and timing scripts of the files above, their environments, summaries of their results, and a README with the commands in order.
REPRODUCE.md		Sources, environments, steps and where each number comes from.
LICENSE, NOTICE, SHA256SUMS		The Apache License 2.0 (the source repository's file), attribution, and the SHA-256 of every other file.

Each row graph holds the whole text decoder, from the embeddings to the final norm: 13.85 to 13.94 GB per file, 75.5 GB for the repository. A graph computes every position of its length whatever the request's length, so the host takes the smallest graph that holds the request. The host uses the files that are present, so you can download only the lengths and tiles you need: a request longer than the largest row graph present is refused (RequestTooLong), and a picture at a tile whose tower is absent too (UnsupportedMedia).

What you need

• A Mac with about 70 GB of free memory for Metal, or about 33 GB for the CPU path. It was measured on a 128 GB Apple M4 Max only.
• About 14 GB of disk per row graph you download (75.5 GB for the whole repository), plus 5 GB of tables, head and tokenizer.
• Python with the packages of host/requirements-host.txt (tested with Python 3.14.6; the host's own test also ran with 3.12.11).
• One to three minutes per row graph, the first time a request needs it: that is the graph's compile.

Minimal usage

Python: desktop GPU or CPU

pip install -U huggingface_hub   # provides the hf command
hf download litert-community/clef-flash-LiteRT --local-dir clef-flash-LiteRT
cd clef-flash-LiteRT
python3 -m venv .venv
.venv/bin/pip install -r host/requirements-host.txt
.venv/bin/python examples/run_example.py --check                     # GPU at float32 precision (Metal on a Mac), about 5 min
.venv/bin/python examples/run_example.py --check --accelerator cpu   # CPU (XNNPACK, 8 threads), about 2 min
.venv/bin/python examples/run_example.py --request examples/request.json --image fixtures/images/clock_3oclock_256x240.png --tile 256

The download takes all 75.5 GB, and --check expects the whole repository: its answers use the 256- and 512-token row graphs and the 256 px vision tower, and its refusal test expects the message of the 4,096-token graph. With fewer files, --request still works for the requests they hold.

The host needs numpy, tokenizers, pillow and ai-edge-litert (2.2.0), pinned with their dependencies in host/requirements-host.txt. It does not need PyTorch or transformers. It was tested on macOS 27.0.

--check answers four public fixture requests (a SemIf record, a support ticket with noul, choice and score questions, a JSON order and a drawn picture) and compares them with examples/run_example.expected.json, the provider's answers in float32 on the CPU. It then sends twelve requests the host refuses and checks each error's name. It prints one line, PASS or FAIL. On the Apple M4 Max it passed on Metal at float32 (max |Δp| 8.71e-5) in 292 s and on the CPU (8.75e-5) in 100 s, compiles included, with a peak memory footprint of 68.1 GB and 33.1 GB. --request answers your own request and prints the response; --image adds one picture.

In your own code:

import sys
sys.path.insert(0, "host")  # run from the repository root
import clef_litert

request = {
    "model": "clef-flash",
    "state": "The parcel came two days late and the box was crushed, but the lamp inside works.",
    "questions": {
        "late": {"type": "noul", "instructions": "Did the parcel arrive late?"},
        "team": {"type": "choice", "instructions": "Which team should handle this?",
                 "criteria": {"shipping": "Deliveries and couriers", "returns": "Replacements and refunds"}},
    },
}

with clef_litert.ClefLiteRT.from_dir(".", accelerator="gpu") as clef:  # accelerator="cpu": XNNPACK, 8 threads
    print(clef.decide(request))

ClefLiteRT.from_dir() builds the host from the files of the folder. It compiles a graph the first time a request needs it and keeps one row graph compiled at a time: a request of another length closes it and compiles the next. On Metal, compiling a row graph took one to three minutes in our runs (59 to 187 s). accelerator="gpu" sets GpuOptions(enforce_f32=True). A request with a picture carries it as "images": [path], read at the 448 px tile by default (decide(request, tile=256) for the 256 px one). A request the host refuses raises a clef_litert.RequestError with the error's name.

Host contract

You need this section only to write your own host: host/clef_litert does all of it. It implements the contract with the provider's defaults (joint_schema_model.py at the revision below), and contract.json holds the same rules as data.

1. Check the request as systemone() does: MissingModelOrState, NoQuestions, BadQuestionType, EmptyCriteria, and SchemaTooLong when the request without its state is over 16,384 tokens. The host also refuses more than one picture, videos, media_kwargs, a picture it cannot open, a URL (UnsupportedMedia), and a request of more than 4,096 tokens (RequestTooLong), never cut.
2. Encode the record as the provider's encode_record(): a fixed prefix, the picture's tokens (<|vision_start|>, one <|image_pad|> per picture token, <|vision_end|> and a newline), the state, the schema of every question, and a fixed suffix, each part tokenized with tokenizer/tokenizer.json and no special tokens.
3. Take the smallest row graph that holds the request and pad it on the right.
4. Write embeds float32 [1, L, 4096]: the rows of tables/embed_tokens.safetensors at the request's ids, widened from bfloat16 to float32 (exact), the picture's rows at its <|image_pad|> positions, and zero on the padding. valid float32 [1, L] is one on the request and zero on the padding.
5. Write cos and sin float32 [1, L, 64]: the multimodal rotary tables of the request's positions (three planes, θ = 10⁷, sections 11 / 11 / 10, interleaved). A text token at index i has position (i, i, i); a picture's tokens take their row and column in the picture's grid; padding continues the positions.
6. Run the row graph's serving_default signature: embeds, valid, cos and sin in, hidden float32 [1, L, 4096] out, the hidden states after the final RMSNorm at every position. No state carries over between calls.
7. Run the joint head on the request's hidden states, in float32 with NumPy: the provider's 2 routing layers and 4 decoder layers (width 1,024, 16 heads) over the request's tokens, with each option's mean lm_head row as its lexical feature. Each question's probabilities are a float32 softmax over its options.
8. Answer as systemone() does: noul gives P(true) rounded to 4 decimals; choice gives choice, confidence and probabilities; score gives score (the expected level), confidence, legend and probabilities. usage counts input_tokens and output_tokens: 0.

Pictures

The host opens a picture with Pillow, converts it to RGB and resizes it to the square tile (BICUBIC), then cuts it into 16 × 16 patches in the merger's order: 256 patches for the 256 px tile and 784 for the 448 px one. The vision tower graph takes patches [1, N, 1536] and the tile's position rows pos [1, N, 1152] from tables/, and returns rows [1, N / 4, 4096], which take the request's <|image_pad|> positions. The provider's own processor reads a picture at its native size; this host reads it at one of the two tiles only. For the checks, the reference resized each picture to the same tile with Pillow (BICUBIC) before the provider's processor read it.

Measured agreement and speed

Agreement with the provider's float32 code

The reference is the provider's code at the revision below, in float32 on the CPU (load_release_model(device="cpu", dtype=torch.float32) and systemone()). The test set has 219 records: 144 SemIf items, 42 requests written for an earlier port of this model, a held-out set of 30 written the same way, and 3 control records. With the pictures read at both tiles, they are 243 runs. A question is a near tie when its two most likely options in the reference are 0.02 or less apart. Each control record changes one word of a question's instructions and must move some option's probability by more than 0.02, which shows that the checks see the input.

The tolerance, set before the runs: the same most likely option on every question that is not a near tie, max |Δp| of at most 0.02 over all options, and a mean over the runs of each run's mean |Δp| of at most 0.002. Each row graph ran a set of the runs it holds on Metal at float32 precision (GpuOptions(enforce_f32=True), every operator on the delegate in one partition) and on the CPU (XNNPACK, 8 threads):

File	Backend	Runs	Same most likely option (not near ties)	Near ties kept	Max |Δp|	Mean |Δp|
L256	Metal	144	144/144	—	1.1e-5	3.6e-7
L256	CPU	144	144/144	—	6.6e-6	4.0e-7
L512	Metal	158	187/187	—	8.0e-6	3.3e-7
L512	CPU	21	50/50	—	6.4e-6	4.4e-7
L1024	Metal	186	322/322	1/1	9.2e-6	3.6e-7
L1024	CPU	186	322/322	1/1	8.6e-6	4.3e-7
L2048	Metal	41	117/117	1/1	6.7e-6	2.5e-7
L2048	CPU	21	85/85	1/1	6.4e-6	4.7e-7
L4096	Metal	42	137/137	—	4.9e-6	2.1e-7
L4096	CPU	22	95/95	—	7.7e-6	4.6e-7

"—": that set held no near tie. End to end, through the host and the files of this repository, the 243 runs on Metal gave 545/545 and the 5 near ties, max |Δp| 1.65e-3 and mean 7.3e-6; 69 of them on the CPU gave 207/207 and 3 near ties, max 1.62e-3. The responses had the reference's structure and decisions on all 243 runs; 200 matched it character for character, and the others differ in some of their four-decimal probabilities (158 of 2,210 values). The 3 control records moved by 0.907, 0.036 and 0.975.

The largest differences come from pictures: the vision towers store their weights in float16. With float32 tower weights, the same 28 picture runs stay within 5.1e-5. Text alone stays within 1.08e-5.

Desktop speed (Apple M4 Max)

These times come from an Apple M4 Max (128 GB, macOS 27.0) with ai-edge-litert 2.2.0 through the Python CompiledModel API. A time is the median of 20 requests after 5 warm-up requests. A request's time covers the host's work, the graph calls (writing the inputs, running, reading the output back) and the joint head.

Request	Row graph	Metal, float32	CPU, 8 threads
Text, 236 tokens	L256	360.7 ms	1,276.5 ms
Text, 459 tokens	L512	673.3 ms	—
Text, 1,007 tokens	L1024	1,346.9 ms	4,726.2 ms
Text, 2,037 tokens	L2048	2,742.9 ms	—
Text, 4,083 tokens	L4096	5,687.2 ms	—
One photo at the 256 px tile, 556 tokens	g256 tower, L1024	1,410.7 ms	—
One photo at the 448 px tile, 688 tokens	g448 tower, L1024	1,493.8 ms	—

One call of each row graph on Metal at float32: 344.1 ms (L256), 654.4 ms (L512), 1,308.5 ms (L1024), 2,707.0 ms (L2048) and 5,570.7 ms (L4096). On the CPU: 1,288.9, 2,407.3, 4,744.5, 9,570.8 and 20,090.4 ms. The joint head alone took 15.3 ms at 236 tokens and 85.1 ms at 4,083 tokens. A vision tower call took 28.9 ms (256 px) and 110.1 ms (448 px) on Metal, 101.0 and 324.6 ms on the CPU. "—": not measured. The Mac was shared with other work: the load average before a measurement ranged from 3.5 to 60.

Each compiled row graph needs desktop memory. On Metal at float32, the process footprint was 34.0 to 38.6 GB after compiling one row graph and peaked at 67.5 to 69.2 GB during the compile (70.2 GB in the example run with a picture); on the CPU it was 28.3 to 28.5 GB.

GPU precision

Run the GPU at float32 precision: the host sets GpuOptions(enforce_f32=True) for accelerator="gpu". At the Metal GPU's default precision (float16 storage), the 256-token row graph moves the probabilities outside the tolerance: max |Δp| 0.253, 3 questions change their most likely option, and many positions come out as zero rows. With its norm inputs pre-scaled, no row is zero, but max |Δp| is still 0.049 and the mean 2.9e-3. The vision towers at that precision return wrong rows. The default precision is only 10 % faster (0.90 times the float32 time at 256 tokens) and gives wrong probabilities, so there is nothing to gain.

Other forms measured

These forms were measured on the same Mac and are not in this repository:

• Dynamic int8 weights: max |Δp| 0.398 on the CPU and 0.101 on Metal, outside the tolerance.
• A chunk step that runs 128 positions per call and carries the state (key and value caches of 4,096 positions and the GatedDeltaNet states) to the next call: within the tolerance on Metal and the CPU. On Metal it was faster than the row graphs only for requests just over one of their lengths (0.81 to 0.96 times their time at 582, 1,086, 2,603 and 2,731 tokens), and slower for the others measured (1.41 and 1.49 times at 193 and 475 tokens).
• A shared-state pair, a state of up to 128 tokens run once and then a questions part of up to 128 tokens: within the tolerance, 1.90 times the row graph's time for one request on Metal and 0.90 times for a second request on the same state.

Limits

• The model never generates text. A request holds at most 4,096 tokens, picture included; a longer one is refused with RequestTooLong, never cut. The provider's code reads up to 16,384 tokens.
• One picture per request, read at a 256 px or 448 px square tile. The provider's native picture size, several pictures and videos are refused.
• Run the GPU at float32 precision. Its default precision is outside the tolerance (see GPU precision).
• Each compiled row graph needs 34 to 39 GB of process footprint on Metal after its compile and about 70 GB during it. The host keeps one row graph compiled at a time; a second process with another graph needs that memory again.
• The probabilities differ from the provider's float32 code by up to 1.08e-5 on text and 1.65e-3 with a picture on the Mac. A question whose two most likely options are closer than that can change its answer; the 5 near ties of the test set did not.
• The agreement numbers measure how closely this conversion follows the provider's code, not task accuracy. Accuracy and calibration were not measured again; see the source model card. The test requests are in English.
• Measured on one Mac (Apple M4 Max) only. GPUs on Linux or Windows and phones were not measured.

Provenance, conversion and license

Details are in REPRODUCE.md and conversion/README.md.

• Source: Cloudflare/clef-flash at revision f0b7d3cd65296abfe459e6c228b618f5d8b26d2f. Base: Qwen/Qwen3.5-9B. Both are licensed under the Apache License 2.0.
• Model: the Qwen3.5 text decoder (32 layers: 24 GatedDeltaNet and 8 gated full-attention layers with 16 query and 4 key-value heads of 256; hidden size 4,096; a 248,320-token vocabulary with untied input and output embeddings), a vision tower of 27 layers (width 1,152) with a 2 × 2 merger to 4,096, and the provider's joint head.
• Row graphs: the Qwen3.5 text model of transformers 5.14.1 with the checkpoint's weights, re-expressed for export: the GatedDeltaNet recurrence as a chunked kernel of 64 positions, the attention as batched matrix products with an additive mask and the grouped keys and values copied by concat, and the rotary tables as inputs. Exported with litert-torch 0.9.4 from the weights in float32, then quantized with ai-edge-quantizer 0.9.0: the 296 FULLY_CONNECTED operators read 250 float16 weight buffers through DEQUANTIZE, one per linear layer (248) and two constant matrices of the GatedDeltaNet kernel that the other 48 share. Activations stay in float32. No operator is CUSTOM, no tensor is int64 and no tensor has a rank above 4.
• Vision towers: the vision tower and merger at one fixed tile, with the position rows as a graph input; float16 FULLY_CONNECTED weights. A form that added the position rows as a constant to a FULLY_CONNECTED output computed wrong values on Metal and was replaced.
• Tables: the checkpoint's embed_tokens and lm_head bytes as they are (bfloat16) under new names, and the vision towers' position rows resampled in float32.
• Scripts: conversion/ holds the conversion, check and timing scripts of the files above, with their environments and a README of the commands; the README lists what is not included.
• Test data: fixtures/ holds the 219 records. The 144 SemIf records are MIT (fixtures/LICENSE-SemIf-MIT.txt). The other records were written for an earlier port of this model; 5 of them are given by the SHA-256 of their request only. The 18 drawn pictures are CC0-1.0, and the 6 photographs are CC0 files from Wikimedia Commons (fixtures/photos.json names each one's page and author). fixtures/reference_probs.json holds the provider's float32 probabilities for every run.

License: clef-flash and Qwen3.5-9B are licensed under the Apache License 2.0 (LICENSE, a copy of the source repository's file). This repository is released under the same license, except the SemIf fixture records (MIT) and the photographs (CC0). Attribution is in NOTICE.

Changes from the original work. The weights were converted from the checkpoint's safetensors (bfloat16) to LiteRT flatbuffers: float16 FULLY_CONNECTED weights, float32 for the rest. The text decoder was re-expressed for export as above and split into row graphs of fixed lengths; the vision tower runs at two fixed tiles. The embedding tables ship separately and the host writes their rows. The joint head and the tokenizer are unchanged. The host ports the provider's request handling, joint head and picture preprocessing to NumPy. This repository is a community conversion and is not affiliated with Cloudflare.