Instructions to use zhandrew/ai-kvm-grounder-8b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use zhandrew/ai-kvm-grounder-8b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="zhandrew/ai-kvm-grounder-8b")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("zhandrew/ai-kvm-grounder-8b")
model = AutoModelForMultimodalLM.from_pretrained("zhandrew/ai-kvm-grounder-8b", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use zhandrew/ai-kvm-grounder-8b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "zhandrew/ai-kvm-grounder-8b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "zhandrew/ai-kvm-grounder-8b",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/zhandrew/ai-kvm-grounder-8b

SGLang

How to use zhandrew/ai-kvm-grounder-8b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "zhandrew/ai-kvm-grounder-8b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "zhandrew/ai-kvm-grounder-8b",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "zhandrew/ai-kvm-grounder-8b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "zhandrew/ai-kvm-grounder-8b",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use zhandrew/ai-kvm-grounder-8b with Docker Model Runner:
```
docker model run hf.co/zhandrew/ai-kvm-grounder-8b
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

ai-kvm GUI grounder (GUI-Owl-1.5-8B, LoRA-merged)

The fast click grounder for the ai-kvm agent. Input: a screenshot and a short target description, the way the planner names a click target. Output: one point on a 0-1000 grid. It answers with about 17 tokens and takes about 134 ms p50 on one H200 at one request at a time.

This is mPLUG/GUI-Owl-1.5-8B-Instruct with a rank-16 LoRA merged in. The LoRA was trained for one epoch at LR 5e-5, with the vision tower and projector frozen and images capped at 1,003,520 pixels.

Prompt (use exactly this)

User turn: the image, then

Locate this element in the screenshot: {target}
Reply with its center as {"point_2d": [x, y]}, integers 0-1000 relative to the image width and height.

Reply: {"point_2d": [x, y]}.

x and y are relative to the image you sent: x_px = x / 1000 * width.
Use no system prompt, temperature 0 and max_tokens 32.
vLLM structured outputs with the JSON schema in shared/ground.py guarantee the reply parses.

Serve

vllm serve <this repo> --max-model-len 8192 --trust-remote-code \
  --limit-mm-per-prompt '{"image":1,"video":0}' \
  --mm-processor-kwargs '{"size":{"longest_edge":1003520,"shortest_edge":65536}}'

In the agent: python3 -m agent.run --grounder http://HOST:PORT/v1 --grounder-model <served name> --grounder-format point "task".

Results

Accuracy is on ScreenSpot-v2 (desktop + web, 771 samples; screenshots downscaled to a 1280 px long side, never trained on). Latency is on an H200, measured at one request at a time over 150 samples.

Model	All	macOS	Windows	Web	Desktop icons	Web icons	p50 latency
GUI-Owl-1.5-8B (base, same prompt)	90.5	92.5	90.6	90.6	80.7	88.2	140 ms
This model	89.6	93.2	94.3	87.4	85.7	82.3	134 ms
Qwen3.8-27B-FP8 (teacher)	94.9	95.2	98.7	93.1	95.7	90.1	275 ms

Desktop, and desktop icons in particular, improve over the base. Web icons regress, so prefer the base model for browser-heavy work until the next iteration fixes this.

Training data (about 11k samples)

Every label is the center of a ground-truth element box. No coordinate comes from a model.

OS-Atlas macOS (Apache-2.0). Text targets kept only when the teacher's click landed inside the box. 20% of the boxes failed that check: menu-bar items had shifted boxes, and labels such as slider values and object reprs were removed.
Live web pages with DOM boxes, checked the same way.
GroundCUA (MIT, human-verified desktop apps). Targets the base model misses had to pass the teacher check; only 43% did, because most were ambiguous.
Teacher-written planner-style descriptions (Qwen3.8-27B looks at the screenshot with the box drawn). These were kept only when the teacher, re-grounding its own description on the clean screenshot, landed inside the box (86% kept).
Difficulty weighting. Targets the base model already hits make up about 35% of the data, as replay. 20% of the data is extra randomly downscaled copies of the screenshots.

The pipeline code is in the ai-kvm repo: training/build_grounding.py, training/describe.py, training/eval_point.py and training/lora_ground8b.yaml.

Downloads last month: 33

Safetensors

Model size

9B params

Tensor type

BF16

Model tree for zhandrew/ai-kvm-grounder-8b

Base model

mPLUG/GUI-Owl-1.5-8B-Instruct

Finetuned

(5)

this model

Quantizations

1 model