Instructions to use talisma/Qwen3-VL-8B-Instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use talisma/Qwen3-VL-8B-Instruct with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf talisma/Qwen3-VL-8B-Instruct:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf talisma/Qwen3-VL-8B-Instruct:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf talisma/Qwen3-VL-8B-Instruct:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf talisma/Qwen3-VL-8B-Instruct:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf talisma/Qwen3-VL-8B-Instruct:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf talisma/Qwen3-VL-8B-Instruct:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf talisma/Qwen3-VL-8B-Instruct:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf talisma/Qwen3-VL-8B-Instruct:Q4_K_M

Use Docker

docker model run hf.co/talisma/Qwen3-VL-8B-Instruct:Q4_K_M

LM Studio
Jan

vLLM

How to use talisma/Qwen3-VL-8B-Instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "talisma/Qwen3-VL-8B-Instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "talisma/Qwen3-VL-8B-Instruct",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/talisma/Qwen3-VL-8B-Instruct:Q4_K_M

Ollama
How to use talisma/Qwen3-VL-8B-Instruct with Ollama:
```
ollama run hf.co/talisma/Qwen3-VL-8B-Instruct:Q4_K_M
```
Unsloth Desktop

How to use talisma/Qwen3-VL-8B-Instruct with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf talisma/Qwen3-VL-8B-Instruct:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "talisma/Qwen3-VL-8B-Instruct:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use talisma/Qwen3-VL-8B-Instruct with Docker Model Runner:
```
docker model run hf.co/talisma/Qwen3-VL-8B-Instruct:Q4_K_M
```

Lemonade

How to use talisma/Qwen3-VL-8B-Instruct with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull talisma/Qwen3-VL-8B-Instruct:Q4_K_M

Run and chat with the model

lemonade run user.Qwen3-VL-8B-Instruct-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use talisma/Qwen3-VL-8B-Instruct with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf talisma/Qwen3-VL-8B-Instruct:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default talisma/Qwen3-VL-8B-Instruct:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use talisma/Qwen3-VL-8B-Instruct with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf talisma/Qwen3-VL-8B-Instruct:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "talisma/Qwen3-VL-8B-Instruct:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

👁️‍🗨️ Qwen3-VL-8B-Instruct — GGUF Edition

An AI that can read, see, and think — now shrunk down to run on your own machine.

In a hurry? Jump to 🧊 Quantizations & Hardware to pick a version that fits your machine, or 🚀 Quick Start to start chatting.

🤔 What am I looking at?

Think of Qwen3-VL as a language model that also has eyes. You can show it a photo, a screenshot, a chart, a scanned document, or even a video, and talk to it about what it sees — the same way you'd chat with a normal AI assistant, just with pictures added to the conversation.

This particular page is for Qwen3-VL-8B-Instruct — a mid-sized version of the model that's small enough to run on a good laptop or desktop, but still genuinely capable.

🏭 What's happening in this repository?

This is a Talisma AI Studio conversion repository. We take the original model — Qwen/Qwen3-VL-8B-Instruct — and:

🔄 Convert it from its original format into GGUF

🗜️ Quantize it into several smaller sizes

👁️ Extract and package the vision encoder (mmproj) separately

☁️ Upload everything here for the team to download and run locally

GGUF + quantization is what makes it possible to run an 8-billion-parameter vision model on ordinary hardware instead of a server rack.

🌟 Why this model is a big deal

Qwen's team describes Qwen3-VL as the most capable vision-language model in the Qwen family so far. In plain terms, here's what changed:

🤖 It can operate apps for you Recognizes buttons, menus, and UI elements on a screen well enough to click through tasks on its own — a real step toward "AI that uses your computer."

🎨 It can turn pictures into code Show it a screenshot or a hand-drawn diagram, and it can generate the matching Draw.io, HTML, CSS, or JS.

📐 It understands where things are Judges object position, viewpoint, and occlusion — not just "what" is in the image, but "where" and "how it's arranged," including in 3D.

📚 It has a very long memory Natively handles 256K tokens of context (expandable to 1M) — enough to recall details across an entire book or an hours-long video.

🧮 It's better at STEM reasoning Stronger at math and science problems that require step-by-step, evidence-based reasoning.

🔤 It reads more languages, more reliably OCR now covers 32 languages (up from 19), and holds up better in blur, tilt, low light, and messy handwriting.

🧠 Curious about the architecture? (click to expand)

Three technical upgrades power the improvements above:

Upgrade	In plain English
Interleaved-MRoPE	A smarter way of encoding time, width, and height together, so the model tracks what's happening across a long video without losing the thread.
DeepStack	Blends visual detail from multiple layers of the vision encoder, so fine details in an image line up more precisely with the words describing them.
Text–Timestamp Alignment	Lets the model point to the exact moment something happens in a video, instead of a rough guess.

🧬 Model Specifications

Property	Value
Base Model	Qwen3-VL-8B-Instruct
Developer	Qwen Team, Alibaba Cloud
Architecture	Qwen3VL (Dense)
Parameters	8B
Modality	Text + Image + Video → Text
Native Context	256K tokens (expandable to 1M)
License	Apache 2.0
This repo provides	GGUF-quantized versions + vision encoder (mmproj)

📊 How good is it, really?

Qwen benchmarked Qwen3-VL-8B-Instruct across a wide range of categories — both multimodal (image/video + text) and pure text tasks — comparing it against the smaller Qwen3-VL-4B and other models in its class.

📈 What gets tested (click to expand)

Multimodal categories: STEM & puzzles, general visual Q&A, subjective/instruction-following, OCR & document/chart understanding, 2D/3D grounding, multi-image reasoning, embodied & spatial understanding, video understanding, agentic (GUI) tasks, and fine-grained perception.

Pure text categories: general knowledge, reasoning, coding, instruction following, subjective evaluation, agentic tool use, and multilingual tasks.

Across both sets, the 8B Instruct model consistently scores higher than the smaller 4B version, and holds its own against other models in its size class — particularly in OCR, document understanding, and agentic/GUI tasks, which were a major focus of this release.

For the full, exact benchmark numbers (they're extensive), see the official model card.

🧊 Quantizations & Hardware

Status: conversion in progress. The table below reflects our planned quantization lineup and typical size/memory expectations for an 8B dense model — not yet Talisma-measured numbers. Once files are uploaded, this section will be updated with real file sizes and confirmed hardware guidance.

Quant	Typical Size	Good For	Quality
`Q2_K`	~3.0 GB	Very constrained devices	🔴 Noticeable quality loss
`Q3_K_M`	~3.8 GB	Lightweight/edge use	🟠 High quality loss
`Q4_K_M`	~4.9 GB	⭐ Everyday laptops	🟡 Balanced
`Q5_K_M`	~5.7 GB	⭐ Best all-round pick	🟢 Low quality loss
`Q6_K`	~6.5 GB	Desktops with headroom	🔵 Very low loss
`Q8_0`	~8.5 GB	High-end machines	⚪ Near-lossless
`F16`	~16 GB	Reference / benchmarking	⚪ Full precision

The vision encoder (mmproj) is packaged and quantized separately from the language model — commonly at Q8_0 or F16 — since it's much smaller and keeping it higher-precision preserves visual accuracy even when the language model itself is more heavily compressed.

Smaller & Faster  ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░  Bigger & Sharper
                  Q2_K   Q3_K_M   Q4_K_M   Q5_K_M   Q6_K   Q8_0   F16

*Most people should start with Q4_K_M or Q5_K_M.*

💻 Not sure what quant/RAM means? (click to expand)

Think of quantization like saving a photo at different compression levels. A Q4 file is like a smaller JPEG — noticeably smaller, and almost as good to the eye. A F16 file is like the original, uncompressed photo — perfect quality, but heavy to store and slow to load. Most people can't tell the difference between Q5 and full quality in everyday use, but the file size difference is huge.

Rule of thumb: your available RAM should comfortably exceed the model file size — actual usage also depends on context length, image size, and how many things you're running at once.

🚀 Quick Start

🐍 Python (Transformers) — for the original model, useful for testing/reference

pip install git+https://github.com/huggingface/transformers

from transformers import Qwen3VLForConditionalGeneration, AutoProcessor

model = Qwen3VLForConditionalGeneration.from_pretrained(
    "Qwen/Qwen3-VL-8B-Instruct", dtype="auto", device_map="auto"
)
processor = AutoProcessor.from_pretrained("Qwen/Qwen3-VL-8B-Instruct")

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg"},
            {"type": "text", "text": "Describe this image."},
        ],
    }
]

inputs = processor.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True,
    return_dict=True, return_tensors="pt"
).to(model.device)

generated_ids = model.generate(**inputs, max_new_tokens=128)
trimmed = [out[len(inp):] for inp, out in zip(inputs.input_ids, generated_ids)]
print(processor.batch_decode(trimmed, skip_special_tokens=True))

💡 For multi-image or video inputs, enabling flash_attention_2 is recommended for speed and memory savings.

🦙 llama.cpp — once GGUF files are uploaded

llama-mtmd-cli \
  -m Qwen3VL-8B-Instruct-Q4_K_M.gguf \
  --mmproj mmproj-Qwen3VL-8B-Instruct-F16.gguf \
  --image photo.jpg \
  -p "What's going on in this picture?"

💻 LM Studio / Open WebUI

Once uploaded here, download the language model file and its matching mmproj file, load them together, and chat as normal — attach an image whenever you like.

🎛️ Generation Settings

Different starting points depending on whether you're mostly using images/video, or pure text:

Setting	Vision-Language (VL)	Pure Text
`temperature`	0.7	1.0
`top_p`	0.8	1.0
`top_k`	20	40
`repetition_penalty`	1.0	1.0
`presence_penalty`	1.5	2.0
`max output length`	16,384	32,768

(greedy=false for both — the model samples rather than always picking the single most likely next word.)

🎯 What can I actually use this for?

No jargon — just everyday examples of what this model is good at:


📸 "What's in this photo?" — captioning & general Q&A	🧾 "Read me this receipt" — OCR & document parsing
📊 "Explain this chart" — chart & graph understanding	🖥️ "What's on my screen?" — screenshot & UI understanding
🎬 "Summarize this video" — long-video comprehension	✍️ "Turn this sketch into code" — visual coding
🧑‍💻 Coding help — text-only or image-grounded	🌍 Reading text in 32 languages — multilingual OCR
🤖 Automating repetitive on-screen tasks — visual agent	📚 Digging through a huge document or long video — long context

🚧 Things to keep in mind

It can be confidently wrong — always double-check important facts, especially in specialized domains.

Quantized versions trade a little quality for size — the more compressed the file, the more small errors can creep in.

Video and multi-image inputs need more memory and time than a single image or plain text.

Not a substitute for professional judgment — avoid relying on it alone for medical, legal, financial, or safety-critical decisions.

🗂️ What's in this repository

File (planned)	Purpose
`README.md`	This page.
`Qwen3VL-8B-Instruct-*.gguf`	Quantized language model, in multiple sizes.
`mmproj-Qwen3VL-8B-Instruct-*.gguf`	Vision encoder — required for image/video understanding.
`download.sh`	Talisma's helper script to fetch model + projector files.
`verify.sh`	Checks downloaded files are complete and intact.

📚 References & Citation

📖 BibTeX citation

@misc{qwen3technicalreport,
      title={Qwen3 Technical Report},
      author={Qwen Team},
      year={2025},
      eprint={2505.09388},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2505.09388},
}

License: Apache 2.0 · Original model by the Qwen Team · GGUF conversion & quantization by Talisma AI Studio

Built for the Talisma AI Studio Model Registry 🧠