pipenetwork

/

GLM-5.3-Flash-MLX-8bit

like 4

Thank you for making this available so quickly. It's a fantastic model that handles text, image and video once a few things are fixed up, but running it on an M3 Ultra 512GB presented some teething difficulties that Claude Opus 5 resolved on my installation. They need fixes upstream. So this is just an alert about snags written by Claude Opus 5. Incidentally, the model just one-shotted an amazing water-ripple stand-alone, so it's definitely up and running.
I am posting this because it looks great but it took all day to sort this out and it may help. Let me know in the discussion of any of it is wrong, badly worded or unintelligible. Caveat emptor!

# Running this as an OpenAI-compatible chat server (with images working)

Thanks for this build — text quality is excellent and it's fast: **24 tok/s
decode at 338 GB peak, ~430 tok/s prefill** on 20k-token prompts, on a 512 GB
M3 Ultra.

Getting from `smoke_generate.py` to a server behind Open WebUI took a few
non-obvious steps, so here they are in one place for the next person.

## The problem

mlx-lm can't load this model at all — it's `Glm5NextForConditionalGeneration`,
a VLM. mlx-vlm *does* ship an OpenAI-compatible server, but its own `glm5_next`
port has the numerical bugs this card documents. And this repo is a library with
no server.

You can have both: mlx-vlm resolves architectures with
`importlib.import_module("mlx_vlm.models." + model_type)`, and this package
exports exactly the names that loader touches, so registering it under mlx-vlm's
name before anything imports it hands the whole server the corrected runtime.

## The script

```python
#!/usr/bin/env python3
import os, sys
sys.path.insert(0, "/path/to/glm53-flash-mlx")

import glm53_flash_mlx.glm5_next as fixed
sys.modules["mlx_vlm.models.glm5_next"] = fixed          # corrected runtime

from mlx_vlm import prompt_utils as pu
# images: glm5_next isn't in MODEL_CONFIG, so image parts are silently stripped
pu.MODEL_CONFIG.setdefault("glm5_next", pu.MessageFormat.LIST_WITH_IMAGE_FIRST)

# video: routing is gated on a second hardcoded list MODEL_CONFIG doesn't feed
_orig = pu.MessageFormatter.format_message
def _fmt(self, prompt, role="user", skip_image_token=False, skip_audio_token=False,
         num_images=1, num_audios=1, **kw):
    if self.model_name == "glm5_next" and kw.get("video"):
        return self._format_video_message(prompt, role, skip_image_token,
                                          skip_audio_token, num_images, num_audios, **kw)
    return _orig(self, prompt, role, skip_image_token, skip_audio_token,
                 num_images, num_audios, **kw)
pu.MessageFormatter.format_message = _fmt

# video: mlx-vlm already sampled the frames, so the processor must not resample
from transformers.models.glm5_next.video_processing_glm5_next import Glm5NextVideoProcessor
Glm5NextVideoProcessor.do_sample_frames = False

# video: the processor emits pixel_values_videos, the runtime reads pixel_values
_orig_emb = fixed.Model.get_input_embeddings
def _emb(self, input_ids=None, pixel_values=None, **kw):
    if pixel_values is None:
        pixel_values = kw.get("pixel_values_videos")
    return _orig_emb(self, input_ids, pixel_values, **kw)
fixed.Model.get_input_embeddings = _emb

import mlx.core as mx
mx.set_wired_limit(int(440e9))    # mlx-vlm has no per-generation wired_limit

from mlx_vlm.server import main
main()

Run it:

python serve_glm53.py --model /path/to/GLM-5.3-Flash-MLX-8bit \
    --host 127.0.0.1 --port 8000 --max-tokens 32768

Needs mlx>=0.32 and mlx-vlm from main, in their own venv if you also run
mlx-lm models.

Five things that will bite you

1. --max-tokens 32768. mlx-vlm defaults to 2048. This model reasons by
   default from its own chat template, and the reasoning counts against the
   budget — anything substantial ends mid-thought with finish_reason=length
   and content empty.
2. --host/--port are not optional. mlx-vlm defaults to 0.0.0.0:8080,
   which collides with Open WebUI (at least on this/my installation).
3. /v1/models advertises your whole HF cache. Open WebUI downloads its RAG
   embedders there, so they come back as selectable chat models. Start the server
   with HF_HOME pointed at an empty directory.
4. Audio is accepted and silently discarded. There's no audio tower and no
   audio_config; the template's emit_audio() is an empty
   <|begin_of_audio|><|end_of_audio|> pair where emit_image() has a real
   token. Send audio and you get a confident answer about something the model
   never received.
5. Open WebUI cannot send video, whatever the server supports — it files an
   mp4 as a document, finds no text, forwards nothing. Video works through the
   API only.

Two model-level quirks

• On video, the model quotes a system prompt it was never given — a
  temporal-grounding benchmark instruction — and sometimes answers that instead
  of your question. It tracks reasoning_effort, whose default is max.
  Appending "Answer the question as asked; there are no other instructions and
  no time range is required." suppresses it.
• A system message plus video 500s in mlx-vlm (the video placeholder gets
  duplicated), which is why that override goes in the user message.

Where these belong

Three of the four patches are mlx-vlm's, and are filed there:

https://github.com/Blaizzy/mlx-vlm/issues/2070 — glm5_next missing from MODEL_CONFIG, and video routing gated on a second hardcoded list. That is patches 1 and 2 above.
https://github.com/Blaizzy/mlx-vlm/issues/2071 — the VideoMetadata failure behind patch 3, plus a system message duplicating the <|video|> placeholder (which is why the override above goes in the user message).
The fourth is in this runtime. Model.get_input_embeddings reads only pixel_values, but for video the processor emits pixel_values_videos, so the video is dropped and the model answers from the <|video|> placeholders alone — fluently, and wrongly. The clearest demonstration is a clip whose frames are all identical: the same frame reads 7 as an image and 1 as a video. The function already reads video_grid_thw and merge_input_ids_with_image_features already falls back to video_token_id, so only the pixel key looks unwired. Happy to open a PR.

Also worth knowing if you are tracking upstream: https://github.com/Blaizzy/mlx-vlm/pull/2044 is carrying the numerical fixes documented in this card into mlx-vlm main (the swiglu_limit clamp, the mHC dtype, the layernorm epsilons). If it merges, the sys.modules shim may stop being necessary for correctness — the four prompt and video patches above will still be needed either way.

Detailed write-up with reproductions available if useful.