musk12/apple-fastvlm-cpu-inference-models

FastVLM CPU Inference (Vision Projector + Qwen2 GGUF)

A complete, CPU-only FastVLM inference pipeline: an ONNX vision projector for image encoding, paired with a quantized Qwen2 GGUF language model for text generation. This repo includes the model weights plus the full loading/serving code, so you can run the pipeline end to end without a GPU.

Pipeline overview

Image → [ONNX Vision Projector] → image embeddings ↓ [fastvlm_server (llama.cpp-based C binary)] ↓ Streamed text response (Qwen2 GGUF, Q4_K_M)

1. An image is preprocessed and passed through vision_projector_v1_standalone.onnx (ONNX Runtime, CPU) to produce image embeddings.
2. The embeddings are written to a temporary binary file and handed to a persistent fastvlm_server process (a custom llama.cpp-based binary) that keeps the GGUF language model loaded in memory.
3. The language model generates a response token-by-token, streamed back over HTTP.

Files in this repo

File	Role
vision_projector_v1_standalone.onnx	Vision encoder — converts image → embeddings
fastvlm_qwen2_q4km.gguf	Quantized Qwen2-0.5B language model (Q4_K_M)
fastvlm_server	Compiled C binary (llama.cpp-based) that loads the GGUF model and serves generation over stdin/stdout
libggml.so, libggml-base.so, libggml-cpu.so, libllama.so (+ .0 versions)	Shared libraries required by fastvlm_server at runtime
stream_api.py	FastAPI server tying the vision encoder and LLM server together, exposing /predict

Requirements

• Python 3.9+
• onnxruntime, fastapi, uvicorn, pillow, numpy, python-multipart

pip install onnxruntime fastapi uvicorn pillow numpy python-multipart

Loading and running

Download all files in this repo into one local directory, keeping them all together (stream_api.py looks for the .onnx, .gguf, fastvlm_server, and .so files relative to its own location), then:

python stream_api.py

This starts a FastAPI server on http://0.0.0.0:8000. On startup it:

• loads the ONNX vision encoder via ONNX Runtime (CPU),
• launches fastvlm_server as a persistent subprocess with the GGUF model loaded and kept in memory, so the model isn't reloaded per request.

Endpoints

• GET / — basic status info
• GET /health — reports whether the ONNX session, GGUF file, and server binary are all present and loaded
• POST /predict — send an image + optional prompt, get a streamed text response

Example request:

curl -X POST http://localhost:8000/predict \
  -F "image=@your_image.jpg" \
  -F "prompt=Describe this image in detail." \
  --no-buffer

The response streams as plain text until generation completes.

How it stays fast

fastvlm_server is started once at API startup and kept alive as a long-running subprocess — the GGUF model is loaded into memory a single time. Each /predict request sends the embedding file path and prompt over the process's stdin and reads generated tokens back from stdout until a ---END--- sentinel, avoiding reload cost per request. Requests are serialized (one at a time), since the underlying server is single-threaded.

Eval results

Benchmarks were run on a 6-core Intel CPU, comparing the two input resolution variants of the vision pipeline (512×512 vs 1024×1024).

Resolution	Visual tokens	TTFT	Benchmark	Score
512×512	64	~1.4 s	POPE	80.39%
512×512	64	~1.4 s	GQA	60.12%
1024×1024	256	~5.5–9.6 s	TextVQA	72.21%
1024×1024	256	~5.5–9.6 s	OCRBench v2	45.57%

LLaVA-Wild average TTFT @ 512: 1,382 ms (6-core Intel CPU)

Notes

• 512×512 uses fewer visual tokens (64) and is significantly faster to first token, making it a good fit for latency-sensitive use cases and general VQA-style tasks (POPE, GQA).
• 1024×1024 uses more visual tokens (256) for finer-grained visual detail, at the cost of higher TTFT — better suited for tasks requiring precise text/detail recognition (TextVQA, OCRBench v2).
• TTFT (time to first token) figures are CPU-only, measured on a 6-core Intel CPU, and will vary with hardware.

Base models

Derived from:

• apple/FastVLM-0.5B (vision projector)
• Qwen/Qwen2-0.5B (language backbone, quantized to GGUF)

Related

A hosted Gradio demo of this pipeline is available at: musk12/FastVLM-CPU-Inference-demo