sdkv2/kev-0.8b-vision-mlx

Kev-0.8B with vision, for MLX

Kev-0.8B (a decision model: LoRA + pointer head on Qwen3.5-0.8B-Base) reading frames on Apple Silicon. The base's own vision tower encodes the frames, and their features take the image-token positions in Kev's input. Positions are the base's 3-axis multimodal (M-RoPE) ones. Kev's pointer head turns one forward pass into a probability per answer option, with no text generation.

Kev was trained on text only and its LoRA touches the language model only, so the vision tower is the base's, unchanged. This repository is not a new fine-tune: it is Kev's LoRA folded into the base, written in mlx-vlm's layout, plus Kev's pointer head and the code that runs it. No other adapters are included.

Files

file	what
model.safetensors	the vision tower and the text model, Kev's LoRA merged (W + B·A·α/r in fp32), fp32, mlx-vlm's layout (zero-centred norm weights stored as 1 + w, computed in fp32)
kev_head.safetensors, kev_config.json	Kev's pointer head (q/k weight and bias, fp32), its head size and temperature (2.351), Kev's special tokens, provenance
config.json, preprocessor_config.json, tokenizer*.json, ...	the base's, unchanged
mlx_vision.py	MLXVisionKev: probs(parts, instr, options) over strings and frames
mlx_stream.py	MLXStream: streaming visual state (one pass per frame, attention trimmed to sinks + a window, evicted keys/values archived per frame, recall by query-key score, re-send)

Use

pip install mlx "mlx-vlm==0.7.11" "transformers==5.17.0" "torch==2.8.0" torchvision pillow huggingface_hub \
    "kev @ git+https://github.com/jaredpalmer/kev@2074cc44256200bb37540103fa4572901faa10f3"

import sys
from huggingface_hub import snapshot_download
d = snapshot_download("sdkv2/kev-0.8b-vision-mlx"); sys.path.insert(0, d)
from mlx_vision import MLXVisionKev

vk = MLXVisionKev(d, dtype="bfloat16", vision_dtype="float32")   # recommended; or dtype="float32" (exact) / "bfloat16"
frame = ...   # H x W x 3 uint8 numpy array
p = vk.probs(["The camera now:\n", frame, "\n"], "Which item is in front of the gates?",
             ["the red box", "the blue ball", "nothing"])   # -> [p0, p1, p2]

parts is the state, in order: strings and frames, as many as you like. instr is the question, and options are the answers it may pick from.

The weights are stored in fp32 and the dtype is chosen at load time:

• dtype="float32": the exact setting.
• dtype="bfloat16": every weight is rounded once at load. This is the same arithmetic as merging into the bf16 base (the reload check below gives identical probabilities).
• vision_dtype="float32" with dtype="bfloat16": the vision tower in fp32 and the backbone in bf16.

The torch dependency is kev's own pointer head (a few small fp32 matmuls) and its tokenizer helper; every forward pass with weights runs in MLX. On Linux, MLX runs on the CPU only (no Metal, and no fast bf16 kernel there).

Streaming:

from mlx_stream import MLXStream
s = MLXStream(vk, window=256)
st = s.start("Video log, one frame per step.\n")
for k, f in enumerate(frames):
    s.push(st, [f"[{k}] ", f, "\n"])
top = s.ranked(st, "Which mark was on the sample?", options)[:1]      # archived frames, best first
s.probs(st, "Which mark was on the sample?", options, recall=top)      # or resend=[label, frame, "\n"]

Speed

Measured on an Apple M4 (16 GB, 8-core GPU, macOS 27.0), mlx 0.32.4 and mlx-vlm 0.7.11. One decision is one 512 x 256 frame (128 image tokens) plus a short text state, 286 tokens in all. Medians over 30 decisions.

setting	s / decision
dtype="bfloat16"	0.223
dtype="bfloat16", vision_dtype="float32"	0.231
dtype="float32"	0.282
streaming, bf16 (one frame step + the question)	0.188
PyTorch fp32 on the M4's CPU (no Metal kernels for Gated DeltaNet)	2.448

Parity with the PyTorch reference

The reference is Kev-0.8B in PyTorch fp32 on the CPU: kev's own loader, plus the base's vision tower and multimodal positions from transformers 5.17.0. Each call is scored by the largest change in any option's probability (max |Δp|) and by whether the top answer differs (a "flip").

The calls tested:

• 160 single-frame perception questions on a synthetic sorting-line camera view;
• 20 decisions recorded from an agent using 1 or 4 frames;
• 24 streaming answers.

setting	calls	max |Δp|	mean |Δp|	flips
fp32, perception	160	2.7e-5	4.1e-6	0
fp32, agent decisions	20	1.8e-5	5.3e-6	0
fp32, streaming answers (window, recall, re-send)	24	2.1e-5	7.7e-6	0
bf16 + fp32 vision tower, perception	160	0.025	0.0047	0
bf16 + fp32 vision tower, agent decisions	20	0.0057	0.0030	0
bf16, perception	160	0.073	0.0091	4 (reference margins 0.0045 to 0.10)
bf16, streaming answers	24	0.039	0.0092	0

Other checks:

• 3-axis positions: identical to transformers' get_rope_index on every call.
• Vision features: in fp32 they agree to 2.7e-5 relative L2.
• This export: loaded back, it gives exactly the probabilities (max |Δp| 0.0) of the model built from jaredpalmer/kev-0.8b plus the base, in fp32 and in bf16.

In bf16 the vision tower carries most of the error on frames, which is why the fp32 vision tower is recommended. In a bf16 stream, the recall ranking beyond the top archived frame is not stable, so rank in fp32 when the order matters.

Note for fp32 users of mlx-vlm or mlx-lm with Qwen3.5: both add the 1 of the zero-centred RMSNorm weights at load time, in the snapshot's dtype (bf16). Casting to fp32 afterwards leaves a ~1e-3 per-channel scale error in every norm. The weights here have 1 + w computed in fp32.

Provenance

• Kev: jaredpalmer/kev-0.8b@v1.0 (snapshot bf75a6a8848ea6960ff2ed108d9ed44c2941174f), its LoRA (186 tensors) and head.pt.
• Base: Qwen/Qwen3.5-0.8B-Base at dc7cdfe2ee4154fa7e30f5b51ca41bfa40174e68.
• Converted with kev at 2074cc44256200bb37540103fa4572901faa10f3, mlx 0.32.4, mlx-vlm 0.7.11 and transformers 5.17.0.

Limits

• Zero-shot vision: Kev never saw images in training. How well it answers about frames depends on the task: measure it on yours.
• Pointer head: it scores the options you give it. It does not generate text and does not verify anything.
• bf16: it moves probabilities by up to a few hundredths, and a near-tied top answer can flip. Use fp32 when a decision hangs on a small margin.
• Licence: Apache-2.0, as for Kev-0.8B and Qwen3.5-0.8B-Base.