Mage-VL-Cowork — LiteRT-LM (GUI agent step)

This repository is a conversion of microsoft/Mage-VL-Cowork to LiteRT-LM (.litertlm), made with hf-to-litertlm for Google's LiteRT-LM 0.19.0: on a Mac M4 Max GPU it reads a screenshot and writes a GUI agent's next action in 10.3 s, from process start to exit (2026-11-04).

It ships two files, and the one to use depends on where the vision encoder can run. Mage-VL-Cowork.litertlm puts the decoder and the vision encoder on the GPU: Mac, iPhone 18 Pro, Pixel 11 Pro. Mage-VL-Cowork_vision-fp16.litertlm puts the decoder on the GPU and the vision encoder on the CPU: Galaxy S26, where the vision encoder on the GPU ends the process (see Where it does not run).

Mage-VL-Cowork (Microsoft) is a 4B computer-use agent: a 24-layer vision encoder and a Qwen3-4B text decoder, trained to answer each step with its reasoning in <think>…</think>, a one-sentence Action: line and a <tool_call> that points in coordinates from 0 to 999. Both files take one screenshot per request, stretched to 1920 × 1088 pixels (2,040 image tokens), and hold 4,096 tokens: the agent's system prompt, the screenshot, your instruction, earlier steps as one line each, and the reply.

File	Decoder, vision encoder	What is inside	Size	sha256
Mage-VL-Cowork.litertlm	GPU, GPU	text decoder with int4 weights (blockwise, 128 per block) and a 4,096-token KV cache, run on the GPU with fp16 activations through an exact rotary-position path; int8 embedding table; vision encoder and adapter with fp32 weights, fp32 activations declared for the GPU	3.80 GB	0d922aa53f65b40e5073e034f7c06c098389002ab4ad9c4f99a5f3654dd77b14
Mage-VL-Cowork_vision-fp16.litertlm	GPU, CPU	the same decoder, embedding table and adapter; a vision encoder with fp16 weights for XNNPACK's fp16 path (see The fp16 vision file)	3.20 GB	1e977695d8f46222653233143e95431ccf6e46bfabe2c77073abf2e8ba75c5df

Which backend. On the Mac, use the main file with the decoder and the vision encoder on the GPU: that form passed our 120-screenshot test (see Correctness) and takes 10.3 s per request, against 56–62 s on the Mac's CPU (see Performance). The iPhone 18 Pro takes the same file on Metal: 26.16 s from a cool phone (engine creation not included), against 85.36 s on its CPU. The Galaxy S26 takes the fp16 vision file with the decoder on the GPU and the vision encoder on the CPU: 50.75 s from a cool phone, against 159.7–182.7 s with everything on its CPU, which started hot. The Pixel 11 Pro takes the main file with the decoder and the vision encoder on its GPU: a mean of 106.97 s after a 7-minute rest. Its vision stage took 29.28–35.25 s on the GPU; on the CPU, 79.83 s from the one cool start we have and 151.99–184.28 s from warm starts (see Performance). These four forms each passed the same test. Run one after another, the phones take longer (see Performance).

How to use

cowork_step.py sends one screenshot and one instruction the way the model's own GUI agent does, prints the reply, and converts the click back to your screenshot's pixels:

pip install litert-lm pillow huggingface_hub
hf download litert-community/Mage-VL-Cowork Mage-VL-Cowork.litertlm samples/settings.png --local-dir mvc
curl -LO https://raw.githubusercontent.com/john-rocky/hf-to-litertlm/main/magevlcowork_work/cowork_step.py
python cowork_step.py mvc/Mage-VL-Cowork.litertlm mvc/samples/settings.png "Open the Downloads settings"

On samples/settings.png (a 1920 × 1080 settings window we drew for this card) the Mac GPU answered:

This is the initial state of the task. I can see the Settings window open, with a left sidebar containing several categories like General, Appearance, Downloads, Privacy, Notifications, and About. In the main panel, the General section is currently selected, showing fields such as device name and email address, plus some update-related options and a Save changes button at the bottom right.

To reach the Downloads settings, I need to switch from the General page to the Downloads category in the sidebar. Since Downloads is already visible as a selectable item in the left navigation, clicking it should load the Downloads configuration panel on the right side of the Settings dialog. I’ll do that now so the next screen can focus on the download preferences rather than the general device and email fields. </think>

Action: Click the Downloads item in the Settings sidebar to open the Downloads settings.

<tool_call> {"name": "computer_use", "arguments": {"action": "click", "coordinate": [203, 321], "button": "left"}} </tool_call>

and the script printed [action 'click', point (390, 347) px on 1920x1080], inside the Downloads item.

The fp16 vision file on the same picture, with the vision encoder on the CPU, clicked the same point on the same Mac:

hf download litert-community/Mage-VL-Cowork Mage-VL-Cowork_vision-fp16.litertlm --local-dir mvc
python cowork_step.py mvc/Mage-VL-Cowork_vision-fp16.litertlm mvc/samples/settings.png "Open the Downloads settings" --vision-backend cpu

What the script does, so you can do the same from your own code:

• The picture. It applies the EXIF orientation and stretches the screenshot to 1920 × 1088 (bicubic, no padding) before the runtime sees it, so the model reads the same pixels as in every test on this card.
• The prompt. A system message with the GUI agent's system prompt (below), then one user turn: the screenshot, then the text Please generate the next move according to the UI screenshot, instruction and previous actions., Instruction: … and Previous actions: with None or one line per earlier step (--previous "Step 1: …"). This is the first-step message of the vendor's agent (osworld_eval/mm_agents/mage_gui_agent.py).
• The point. Coordinates are in 0–999 for the whole screen; pixels = round(x / 999 × (width − 1)), round(y / 999 × (height − 1)), as the vendor's agent converts them.
• One screenshot per request. The model card's agent keeps the 3 most recent screenshots; these files have room for one, because a second would take at least another 2,040 tokens. With no earlier steps a request uses 2,986–2,996 of the 4,096 tokens (the longest reply in our test was 460). Each earlier step adds 13–14 tokens, and a request of 4,096 tokens or more is refused (Input token ids are too long: 120 lines made 4,673). Long requests has 40 earlier steps on each device.
• Executing the action is up to your app. The model card asks for an isolated sandbox and a human confirmation before irreversible actions ("Responsible AI & Safe Deployment" in the model card).

--backend and --vision-backend choose the backends (the fp16 vision file needs --vision-backend cpu), and --cache-dir keeps the runtime's caches away from the model file.

The system prompt (verbatim MAGE_GUI_SYSTEM_PROMPT of the vendor's GUI agent; the 120-screenshot test used it)

You are a multi-purpose intelligent assistant. Based on my requests, you can use tools to help me complete various tasks.

# Tools

You have access to the following functions:

<tools>
{"type": "function", "function": {"name": "computer_use", "description": "Use a mouse and keyboard to interact with a computer screen.", "parameters": {"properties": {"action": {"description": "* `click`: Click at (x,y). Use `button` for left/right/middle.\n* `double_click`: Double click at (x,y).\n* `triple_click`: Triple click at (x,y) to select line/paragraph.\n* `move`: Move cursor to (x,y) without clicking.\n* `drag`: Drag from `start` to `end` coordinates.\n* `type`: Type text string.\n* `keypress`: Press key(s). Single key or combo.\n* `scroll`: Scroll at optional (x,y) position. scroll_x/scroll_y in [-999,999], |999|=full screen. Negative scroll_y=down, negative scroll_x=left.\n* `wait`: Wait specified seconds.\n* `terminate`: End the task.", "enum": ["click", "double_click", "triple_click", "move", "drag", "type", "keypress", "scroll", "wait", "terminate"], "type": "string"}, "coordinate": {"description": "(x,y) in [0,999].", "type": "array"}, "button": {"description": "Mouse button for click: left/right/middle.", "type": "string", "enum": ["left", "right", "middle"]}, "start": {"description": "(x,y) drag start in [0,999].", "type": "array"}, "end": {"description": "(x,y) drag end in [0,999].", "type": "array"}, "keys": {"description": "Key list for keypress.", "type": "array"}, "text": {"description": "Text for action=type.", "type": "string"}, "scroll_x": {"description": "Horizontal scroll [-999,999]. Negative=left.", "type": "number"}, "scroll_y": {"description": "Vertical scroll [-999,999]. Negative=down.", "type": "number"}, "time": {"description": "Seconds for wait.", "type": "number"}, "status": {"description": "success/failure for terminate.", "type": "string", "enum": ["success", "failure"]}, "msg_to_user": {"description": "Optional message for terminate.", "type": "string"}}, "required": ["action"], "type": "object"}}}
</tools>

For every step, reply in EXACTLY this format with NO suffix:

<think>
your reasoning paragraph describing the current screen state, how it connects to the task goal, and the rationale for the chosen action
</think>

Action: one short imperative sentence describing the next move.

<tool_call>
{"name": "example_function_name", "arguments": {"example_parameter_1": "value_1"}}
</tool_call>

<IMPORTANT>
Reminder:
- Function calls MUST follow the specified format
- Required parameters MUST be specified
- Reasoning goes inside <think>...</think> BEFORE the Action line; nothing after the last tool_call
- You may output at most 5 <tool_call> blocks per response
- Use `type` for text input, `keypress` for shortcuts and special keys
</IMPORTANT>

# Response format

For every step:
1) <think>...</think> block with your reasoning.
2) Action: a short imperative sentence.
3) One or more <tool_call>...</tool_call> blocks (max 5).

Rules:
- The screen's resolution is 1000x1000. Coordinates are in [0, 999] range.
- Scroll amounts are normalized: |999| = one full screen width/height.
- For scroll, you may include coordinate to scroll at that position.
- For drag, specify start and end coordinates.
- Be brief: one sentence for Action.
- Do not output anything after the last tool call.
- If finishing, use action=terminate.

The model card asks for a longer prompt, MAGE_HYBRID_SYSTEM_PROMPT, which adds a bash tool next to computer_use ("Use the exact system prompt below for best performance", "System Prompt & Tool Schema" in the model card). It fits these files: with its 1,095 tokens (the GUI prompt: 897), a screenshot and an instruction, a request took 3,179–3,196 tokens on 12 test tasks of ours, which leaves 900–917 for the reply, and all 12 replies ended at <|im_end|> (the longest 495 tokens; main file, Mac GPU). cowork_step.py sends the GUI prompt only.

The hybrid prompt's first reply is usually screenshot, so a step takes one or two inferences. Its action list has screenshot, and with Previous actions: None the model usually asks for the screen although the screenshot is in the request: on those 12 tasks the first tool call was {"action": "screenshot"} on all 6 GUI tasks and on 3 of the 6 shell tasks (the other 3 called bash). The model itself does this: the original model in PyTorch (fp32, CPU, the same request) answered screenshot on the one GUI task and bash on the one shell task we ran it on, as the main file did on those two.

An app can answer that call as the vendor's agent does (_build_messages with one image kept). A first reply that calls bash is the step. After a screenshot reply, start a new conversation with the system prompt; a user turn [This screenshot has been collapsed.] and the same text; an assistant turn with the model's reply; a user turn <tool_response>, the screenshot, </tool_response>. The second request took 3,342–3,451 tokens on the test below, and prompt and reply together reached 3,737 at most.

The test: 11 requests on screens we drew (6 GUI, 5 shell), with the expected tool, target and command patterns written down before any reply was read, and none of them used to choose this form. Mage-VL-Cowork_vision-fp16.litertlm, decoder on the GPU, vision encoder on the CPU, the reply of the last inference:

	Right tool	Click inside the target	Tool call parses as JSON	Ended by itself	Command has the expected patterns
Galaxy S26, in an Android app	10 of 11	4 of 6	11 of 11	11 of 11	1 of 5
Mac M4 Max	10 of 11	4 of 6	11 of 11	11 of 11	2 of 5

The two chose the same tool on all 11, and their clicks hit and missed the same targets. The last column is low because many commands are a look around first (ls, pwd, find), which the agent would read before its next step; the patterns ask for the file or folder on the screen and the kind of command that does the job. The Galaxy S26 took 102.8–116.1 s for a step with two inferences and 68.6–74.3 s for a step with one from a cool phone; these 11, on a hot phone, took a median of 219.4 s and 126.8 s, and in another session a two-inference step reached 334.4 s after 17 minutes of steps. On the Pixel 11 Pro we ran this file in the same app for one step of each kind, not for these 11: from thermal status 0, 239.8 s for a step with two inferences, then 171.9 s for a step with one.

The model card's MAGE_HYBRID_SYSTEM_PROMPT (verbatim)

You are a multi-purpose intelligent assistant. Based on my requests, you can use tools to help me complete various tasks.

# Tools

You have access to the following functions:

<tools>
{"type": "function", "function": {"name": "computer_use", "description": "Use a mouse and keyboard to interact with a computer screen.", "parameters": {"properties": {"action": {"description": "* `click`: Click at (x,y). Use `button` for left/right/middle.\n* `double_click`: Double click at (x,y).\n* `triple_click`: Triple click at (x,y) to select line/paragraph.\n* `move`: Move cursor to (x,y) without clicking.\n* `drag`: Drag from `start` to `end` coordinates.\n* `type`: Type text string.\n* `keypress`: Press key(s). Single key or combo.\n* `scroll`: Scroll at optional (x,y) position. scroll_x/scroll_y in [-999,999], |999|=full screen. Negative scroll_y=down, negative scroll_x=left.\n* `wait`: Wait specified seconds.\n* `screenshot`: Capture the current screen without changing anything.\n* `terminate`: End the task.", "enum": ["click", "double_click", "triple_click", "move", "drag", "type", "keypress", "scroll", "wait", "screenshot", "terminate"], "type": "string"}, "coordinate": {"description": "(x,y) in [0,999].", "type": "array"}, "button": {"description": "Mouse button for click: left/right/middle.", "type": "string", "enum": ["left", "right", "middle"]}, "start": {"description": "(x,y) drag start in [0,999].", "type": "array"}, "end": {"description": "(x,y) drag end in [0,999].", "type": "array"}, "keys": {"description": "Key list for keypress.", "type": "array"}, "text": {"description": "Text for action=type.", "type": "string"}, "scroll_x": {"description": "Horizontal scroll [-999,999]. Negative=left.", "type": "number"}, "scroll_y": {"description": "Vertical scroll [-999,999]. Negative=down.", "type": "number"}, "time": {"description": "Seconds for wait.", "type": "number"}, "status": {"description": "success/failure for terminate.", "type": "string", "enum": ["success", "failure"]}, "msg_to_user": {"description": "Optional message for terminate.", "type": "string"}}, "required": ["action"], "type": "object"}}}
{"type": "function", "function": {"name": "bash", "description": "Run a shell command on the machine and read its output. Use it for file inspection and edits, and for anything the shell does more reliably than the GUI; the result comes back as text, not as a screenshot.", "parameters": {"properties": {"command": {"description": "The shell command to run.", "type": "string"}}, "required": ["command"], "type": "object"}}}
</tools>

For every step, reply in EXACTLY this format with NO suffix:

<think>
your reasoning paragraph describing the current screen state, how it connects to the task goal, and the rationale for the chosen action
</think>

Action: one short imperative sentence describing the next move.

<tool_call>
{"name": "example_function_name", "arguments": {"example_parameter_1": "value_1"}}
</tool_call>

<IMPORTANT>
Reminder:
- Function calls MUST follow the specified format
- Required parameters MUST be specified
- Reasoning goes inside <think>...</think> BEFORE the Action line; nothing after the last tool_call
- You may output at most 15 <tool_call> blocks per response
- Use `type` for text input, `keypress` for shortcuts and special keys
- Every tool_call in one response must use the same function
</IMPORTANT>

# Response format

For every step:
1) <think>...</think> block with your reasoning.
2) Action: a short imperative sentence.
3) One or more <tool_call>...</tool_call> blocks (max 15).

Rules:
- The screen's resolution is 1000x1000. Coordinates are in [0, 999] range.
- Scroll amounts are normalized: |999| = one full screen width/height.
- For scroll, you may include coordinate to scroll at that position.
- For drag, specify start and end coordinates.
- Choose `bash` when the shell is the reliable way to do or check something, and `computer_use` when the work has to happen in the interface on screen.
- A `bash` call is answered with the command's text output; a `computer_use` call is answered with a new screenshot.
- Be brief: one sentence for Action.
- Do not output anything after the last tool call.
- If finishing, use action=terminate.

The model card's table of the two tools (verbatim):

Channel	Tool Name	Supported Operations & Return Contract
GUI Channel	computer_use	click, double_click, triple_click, move, drag, type, keypress, scroll, wait, screenshot, terminate. Coordinates are normalized to [0, 999] (x = round(coord_x * width / 1000)). Returns the post-action desktop screenshot.
Terminal Channel	bash	Executes shell commands, Python scripts, and CLI utilities in /tmp_workspace. Returns merged stdout/stderr text inside <tool_response> without consuming an image slot.

(The GUI agent's prompt above has no screenshot action, and the vendor's agent converts points with x / 999 × (width − 1), not the table's x × width / 1000; the two differ by at most 1 pixel on a 1920 × 1080 screen.)

The reply. The template of both files starts the reply with <think> (thinking on, the model's default), and the reply ends at <|im_end|> (id 151645); there is no start token. LiteRT-LM returns the whole reply as text: we passed no tools to the runtime (the system prompt holds them), so read the first <tool_call> block yourself. Read it leniently: with four earlier steps in Previous actions:, the original model wrote “arguments" (a typographic opening quote, which JSON rejects) in 2 of 40 replies; none of the 120 test replies on the Mac GPU, the iPhone, the Pixel 11 Pro or the Galaxy S26 did.

Sampling. The files carry no sampler settings, so LiteRT-LM 0.19.0 samples with top-k 1 (greedy) on the CPU and the GPU; every result on this card is greedy. The model card's own setting is temperature 0.7 and top_p 0.8 (or temperature 0.0 for a deterministic single run).

Correctness

The test: 120 screenshots of ScreenSpot-v2 (40 desktop, 40 web, 40 mobile, taken from each group by a fixed shuffle), each stretched to 1920 × 1088 and sent as the agent's first step with the system prompt above, Instruction: Click on the element for: <the ScreenSpot instruction> and Previous actions: None. A reply counts as right when its first tool call is a click whose point, converted to the screenshot's pixels, falls inside the target's box. The original model was run the same way in PyTorch on the Mac's GPU (MPS; transformers 4.57.3, the same pictures and prompt, greedy). The bar per group is the original model's rate in fp32 or in bf16 (the checkpoint's own type), whichever is lower, minus 5 points.

	Desktop	Web	Mobile	All 120
Mage-VL-Cowork.litertlm, Mac M4 Max GPU	55.0 % (22)	65.0 % (26)	55.0 % (22)	58.3 % (70)
Mage-VL-Cowork.litertlm, iPhone 18 Pro (Metal)	60.0 % (24)	65.0 % (26)	60.0 % (24)	61.7 % (74)
Mage-VL-Cowork.litertlm, Pixel 11 Pro GPU	52.5 % (21)	60.0 % (24)	55.0 % (22)	55.8 % (67)
Mage-VL-Cowork_vision-fp16.litertlm, Galaxy S26 (decoder GPU, vision CPU)	62.5 % (25)	65.0 % (26)	60.0 % (24)	62.5 % (75)
Original model, fp32	52.5 % (21)	65.0 % (26)	62.5 % (25)	60.0 % (72)
Original model, bf16	65.0 % (26)	62.5 % (25)	55.0 % (22)	60.8 % (73)
Bar	47.5 %	57.5 %	50.0 %

The screenshots each form and the original model disagree on (right only in the original / right only in this form, with the exact McNemar p on those two counts):

Form, against	Desktop	Web	Mobile	All 120
Mac GPU, fp32	3 / 4 (1.0)	3 / 3 (1.0)	6 / 3 (0.51)	12 / 10 (0.83)
Mac GPU, bf16	6 / 2 (0.29)	2 / 3 (1.0)	3 / 3 (1.0)	11 / 8 (0.65)
iPhone 18 Pro, fp32	3 / 6 (0.51)	3 / 3 (1.0)	3 / 2 (1.0)	9 / 11 (0.82)
iPhone 18 Pro, bf16	5 / 3 (0.73)	2 / 3 (1.0)	0 / 2 (0.50)	7 / 8 (1.0)
Pixel 11 Pro, fp32	4 / 4 (1.0)	4 / 2 (0.69)	6 / 3 (0.51)	14 / 9 (0.40)
Pixel 11 Pro, bf16	8 / 3 (0.23)	3 / 2 (1.0)	3 / 3 (1.0)	14 / 8 (0.29)
Galaxy S26, fp32	4 / 8 (0.39)	2 / 2 (1.0)	3 / 2 (1.0)	9 / 12 (0.66)
Galaxy S26, bf16	4 / 3 (1.0)	2 / 3 (1.0)	0 / 2 (0.50)	6 / 8 (0.79)

All 120 replies on the Mac GPU, the iPhone, the Pixel 11 Pro and the Galaxy S26 have a tool call that parses as JSON, end at <|im_end|>, and have an Action: line after </think>. LiteRT-LM 0.19.0, one process per screenshot.

• On the Mac GPU and the Pixel 11 Pro, mobile is 7.5 points under the original model in fp32 and equal to it in bf16. On the 40 mobile screenshots the fp32 original is right and the main file wrong on 6, the other way round on 3 (exact McNemar p = 0.51), on each of the two. On the iPhone and the Galaxy S26, mobile is 2.5 points under fp32 and 5 points over bf16. The Pixel 11 Pro's web is 5 points under fp32 and 2.5 under bf16, its 55.8 % of all 120 is the lowest of the four devices, and it clears the bar by 2, 1 and 2 screenshots. Between fp32 and bf16 the original model itself is right on one and wrong on the other on 15 of the 120 screenshots.
• The devices do not give the same reply. The iPhone and the Mac GPU run the same file, and their replies were equal on none of the 120 screenshots (the same click point on 28): both right on 65, only the iPhone on 9, only the Mac on 5, neither on 41 (exact McNemar p = 0.42). The Pixel 11 Pro against the Mac GPU, the same file: equal on none, the same point on 37, both right on 65, only the phone on 2, only the Mac on 5, neither on 48 (p = 0.45). The Galaxy S26, with the fp16 vision encoder, against the Mac GPU: the same point on 22, both right on 64, only the phone on 11, only the Mac on 6, neither on 39. The same screenshot gave the same reply on repeated launches with cold and warm caches on the iPhone (5 of 5) and the Galaxy S26 (3 of 3). On the Pixel 11 Pro the warm replies change from process to process: desktop_006 gave one reply byte for byte on 3 cold processes (inside the target) and 4 different replies on 4 warm ones (3 inside, 1 outside). In the test a session's first process was cold: 5 of those 10 screenshots were right, and 62 of the 110 warm ones. Ten rows are too few to read a difference, and we do not know why the warm replies change.
• Why one 1920 × 1088 picture. The vendor's agent sends 1920 × 1080 desktop screenshots, which the model's own processor turns into a 68 × 120 grid of patches, the grid of 1920 × 1088. At other sizes the original model misplaces its clicks: a drawn button at 9 places was hit 9 of 9 times at 1920 × 1088 and at most 4 of 9 at 1920 × 768, 1920 × 544, 1280 × 736, 960 × 544, 2560 × 1440 and 1088 × 2400; on the 40 desktop screenshots, 1344 × 768 was right on 3 against 21 at 1920 × 1088. So these files take every screenshot stretched to 1920 × 1088, portrait phone screens included.
• Earlier steps as lines. With four generic earlier steps under Previous actions:, the original model (fp32) was right on 20 of the 40 desktop screenshots against 21 without them (exact McNemar p = 1.0).
• 12 of the 120 screenshots on other backends of the Mac. The main file with the decoder and the vision encoder on the CPU (4 threads) clicked inside the target on 7 of the 12, and Mage-VL-Cowork_vision-fp16.litertlm with the decoder on the GPU and the vision encoder on the CPU on 8 (the main file on the GPU: 8). All 24 replies parse as JSON and end at <|im_end|>. On the CPU the reply changes from process to process (the Performance rows: 251, 281 and 312 tokens on one screenshot), so that 7 is one draw. All 120 on the Mac's CPU were run only on an earlier build with a shorter KV cache, not on these files: desktop 67.5 %, web 60.0 %, mobile 57.5 %, 61.7 % of all 120, every reply parsing and ending at <|im_end|> (that build has the same vision encoder, and on the Mac GPU its replies equal the main file's byte for byte on all 120).
• Slips in the reasoning. With earlier steps in the prompt, a function word in the reasoning sometimes comes out wrong (it’t, I9), and the tool call still parses. The original model does it too, so this is not specific to LiteRT or int4: in PyTorch (fp32) with four earlier-step lines, 21 such words in 10,779 reply tokens on 40 screenshots, and none on the same 40 without them. With 40 lines: 1 in 928 tokens on the Mac GPU, 0 in 758 on the iPhone, 3 in 1,023 on the Pixel 11 Pro (all in one reply), 2 in 836 on the Galaxy S26. With no earlier steps: none in the 120 replies on the Mac GPU, the iPhone, the Pixel 11 Pro and the Galaxy S26. A space after an opening quote (“ forget) is different: it appears only in the LiteRT forms (11 on the Mac GPU, 7 on the iPhone, 8 on the Pixel 11 Pro, 11 on the Galaxy S26 in 120 replies each; none in the original model's), and we do not know why.
• The model card's ScreenSpot-v2 figure, 87.2 %, is the result of the first of its five training stages ("In Stage 1 (GUI-oriented midtraining) …"), measured its own way; on this card's test (the agent's step on 120 stretched screenshots) the original model scores 60.0 % in fp32.

Long requests

fp16 holds integers exactly only up to 2,048, so these requests depend on the decoder's rotary path on each device's GPU. We ran the longest requests we have on every device, one process each: the hybrid prompt as written with the picture and text of desktop_006 (3,186 tokens), and three requests with 40 earlier-step lines (3,546–3,547 tokens, on desktop_006, web_013 and mobile_491).

Device, file, backends	Whole prompt prefilled	Ended at the stop token	Tool call parses, Action: line	Largest position (prompt + reply)	Click inside the target (the 3 with earlier steps)
Mac M4 Max, main file, GPU	4 of 4	4 of 4	4 of 4	3,900	1 of 3
iPhone 18 Pro, main file, Metal	4 of 4	4 of 4	4 of 4	3,832	1 of 3
Pixel 11 Pro, main file, GPU	4 of 4	4 of 4	4 of 4	4,027	1 of 3
Galaxy S26, fp16 vision file, decoder GPU + vision CPU	4 of 4	4 of 4	4 of 4	3,875	1 of 3

The reply to the hybrid prompt was the screenshot call on all four. On the Mac GPU the same decoder declared fp32 also clicked inside the target on 1 of the 3.

Performance

One request = one fresh process that creates an Engine, sends one screenshot with the prompt above and streams the reply, with LiteRT-LM 0.19.0: the Python API on the Mac and, through the PyPI Android wheel litert-lm-api in CPython on the phone, on the Galaxy S26 and the Pixel 11 Pro; the iOS release with its Swift API in a benchmark app on the iPhone. The screenshot is ScreenSpot-v2 desktop_006 (2,988 tokens in; out: 267 on the Mac GPU, 251–312 on the Mac CPU, where the reply changes from process to process, 242 on the iPhone's Metal, 261 on its CPU, 206 cold and 229–277 warm on the Pixel 11 Pro's GPU, 261 on the Galaxy S26's decoder GPU, 297 on its CPU), and the rows marked one after another are the 120 test screenshots with no wait for the phone to cool inside a session.

• Request: on the Mac, the process's wall time from start to exit; on the Galaxy S26 and the Pixel 11 Pro, from before creating the Engine to the end of the reply; on the iPhone, from creating the conversation to the end of the reply, so creating the Engine (the next column) comes on top.
• Engine = creating the Engine. Creating the conversation, where the vision executor is built, is in Request and in no column of its own: on the Pixel 11 Pro it took 5.15–8.35 s in the warm and cold rows and a median of 18.07 s one after another. First chunk = from sending the message to the first chunk of the reply (on the iPhone, from creating the conversation). Vision stage, on the two Android phones = first chunk less the runtime's time to first token. Prefill and decode = the runtime's own counters.
• Peak: on the Mac and the iPhone, the process's peak memory footprint (/usr/bin/time -l, phys_footprint). On the Galaxy S26, two figures: GPU = the kgsl page allocation of the whole phone, process = VmHWM. On the Pixel 11 Pro: GPU = the process's GPU memory from dumpsys gpu --gpumem, read on one cold process (6.60 GB after the reply), process = VmHWM.
• Warm = the runtime's cache directory was written by an earlier process; cold = it was empty. From a cool phone: on the iPhone the app waited for thermal state nominal before each launch (167–481 s); on the Galaxy S26 each process started at thermal status 0 or 1, with no CPU cap and the GPU at its top level. After a 7-minute rest, on the Pixel 11 Pro: each warm process waited 421–423 s and started at thermal status 0, with a skin temperature of 34.5–35.8 °C; the cold process opened that session with no such wait, at status 1 and 37.4 °C.
• On the Mac the other heavy jobs were paused, another job's file copy ran during the GPU's one-request rows, and the CPU rows use the runtime's default thread count.

Device, file, backends	State	Request (s)	Engine (s)	First chunk (s)	Prefill (tok/s)	Decode (tok/s)	Peak
Mac M4 Max, main file, GPU	warm (5 processes)	10.3 (10.0–10.5)	0.37–0.39	5.46–5.66	1,199–1,253	87.4–90.5	4.43–4.45 GiB
Mac M4 Max, main file, GPU	cold	12.6	2.34	5.46	1,244	88.6	4.84 GiB
Mac M4 Max, main file, CPU	warm (2 processes)	56.3 / 62.0	0.36 / 0.37	43.0 / 43.9	111 / 104	21.9 / 17.5	3.36 GiB
Mac M4 Max, main file, CPU	cold	57.9	0.72	43.3	113	18.5	3.64 GiB
iPhone 18 Pro, main file, Metal	warm, from a cool phone (3 launches)	26.16 (26.07–26.23)	0.95–0.98	19.27–19.43	513–516	35.2–35.3	3.70 GiB
iPhone 18 Pro, main file, Metal	cold	27.88	2.18	20.99	512	34.2	4.22 GiB
iPhone 18 Pro, main file, Metal	one after another (120, the median)	58.73 (25.24–84.26)	0.91 (0.89–2.31)	44.07 (19.68–60.15)	189 (139–508)	16.3 (12.1–34.2)	3.69 GiB warm, 4.28 GiB at most (cold)
iPhone 18 Pro, main file, CPU	cold	85.36	1.00	60.81	81.5	10.5	3.53 GiB
Pixel 11 Pro, main file, GPU	warm, after a 7-minute rest (3 processes, the mean)	106.97 (102.83–109.75)	8.95–15.42	53.52–56.88	109–126	6.9–8.5	process 1.03 GB
Pixel 11 Pro, main file, GPU	cold	110.43	14.47	59.20	111	7.1	GPU 6.60 GB (another cold process), process 1.32 GB
Pixel 11 Pro, main file, GPU	one after another (120, the median; the mean 147.0)	148.8 (108.10–166.00)	27.87 (12.67–33.79)	64.68 (56.38–75.11)	100 (74–121)	7.0 (5.8–8.2)	process 1.32 GB
Pixel 11 Pro, an earlier build of the fp16 vision file with a shorter KV cache, decoder GPU + vision CPU	warm, one process, from thermal status 1, no 120-screenshot test	140.27	6.28	96.47	137	7.8	process 1.94 GB
Galaxy S26, fp16 vision file, decoder GPU + vision CPU	warm, from a cool phone (2 processes)	50.75 (49.6, 51.9)	4.85, 5.21	30.2, 31.9	263, 234	17.9, 17.3	GPU 4.49 GB, process 2.67 GB
Galaxy S26, fp16 vision file, decoder GPU + vision CPU	cold	51.9	5.68	31.5	275	18.6	GPU 4.49 GB, process 2.51 GB
Galaxy S26, fp16 vision file, decoder GPU + vision CPU	one after another (120, the median; the mean 82.98)	83.8 (56.6–101.6)	10.67 (5.38–12.59)	50.34 (33.9–57.63)	147 (119–257)	10.8 (9.4–18.2)	GPU 4.49 GB, process 2.85 GB
Galaxy S26, fp16 vision file, CPU	warm, started hot	182.7	2.92	122.3	34.1	5.16	process 5.79 GB
Galaxy S26, fp16 vision file, CPU	cold, started hot	159.7	3.89	99.2	41.3	5.33	process 5.65 GB

• One after another depends on more than the file. On the iPhone, 107 of the 120 started and ended at thermal state serious (88 charging, 32 on the battery, that evening and night). An earlier build with a shorter KV cache, whose replies equal the main file's byte for byte on these screenshots, took 45.7–48.3 s at the 6th to 20th request of a session in the daytime (charging, battery at 80 %) and 60.8–68.0 s at the same places that night (charging, battery at 45 %). That night, in the same order, the main file took 1.00, 0.99 and 0.98 times the earlier build's seconds at the 1st to 5th, 6th to 10th and 11th to 15th request (5 requests each, one pass, the earlier build first). We did not separate what moved the day and the night apart; the battery level and the hour differ. On the Galaxy S26, 108 of the 120 started at thermal status 3 (a mean of 85.43 s, against 59.34 s from status 1), and 118 started under a CPU cap. On the Pixel 11 Pro, 92 of the 120 started at thermal status 3 (a mean of 151.3 s, against 115.35 s from status 0 and 120.93 s from status 1), 39 reached status 4, and the skin temperature reached 46.9 °C.
• Memory on the phones. The iPhone's app has the increased-memory-limit entitlement, and the limit it read at launch was 6,144 MiB; at least 2,004 MiB stayed free in every launch of the main file, and we did not run these files without the entitlement. On the Galaxy S26, free memory (MemAvailable) fell by 4.95–5.10 GB in the three cool-phone processes, to 2.21 GB (2,154,068 kB) at the lowest. On the Pixel 11 Pro, a phone with 16 GB, it fell by 7.23–7.47 GB in the warm and cold rows' four processes; the lowest reading in any of the main file's rows was 2.36 GB (2,305,828 kB), on the hybrid prompt's cold process.
• The CPU rows on the phones. The iPhone's is one cold launch: four more launches on the battery went from thermal state nominal to serious, which our timing rule does not keep (76.34–84.88 s), so there is no warm row. The Galaxy S26's two ran right after a GPU run with no wait, at thermal status 3 and under different CPU caps (2.11 GHz cold, 1.23–1.27 GHz warm), so they do not compare the caches. We did not run the decoder on the Pixel 11 Pro's CPU.
• The vision encoder on the Pixel 11 Pro's CPU. We name the GPU for this phone: its vision stage stayed at 29.28–35.25 s from a skin temperature of 34.5–37.4 °C (30.39–37.82 s on the 120, where the skin reached 46.9 °C). On the CPU, one process each on desktop_006, the fp32 encoder took 79.83 s from the one cool start we have (30.6 °C, an earlier build with a shorter KV cache, an empty cache) and 151.99–184.28 s from warm starts (37.0–39.6 °C, that build and the main file, whose requests took 211.57 s warm and 248.02 s cold). Among those warm starts an empty cache took 179.95–184.28 s and a written one 151.99–159.06 s, so the temperature and the cache are both in these numbers. The earlier build of the fp16 vision file in the row above took 74.50 s warm and 92.21 s cold, both from 38.9 °C. Mage-VL-Cowork_vision-fp16.litertlm itself ran on this phone only in the Android app above, on the hybrid prompt: a vision stage of 35.6 s from thermal status 0, and 90.8 s on the next inference. Neither form took the 120-screenshot test on this phone.

Two earlier warm requests on the Mac GPU, with another job's file copy over them, took 11.9 s and 10.1 s: the slower one spent 2.9 s creating the conversation against 1.06–1.32 s in the other warm requests; we do not know why, and it did not recur in the five above.

The decoder alone with litert-lm benchmark -p 1024 -d 256 --runs 3 --cache no on the Mac: GPU prefill 1,352 tok/s, decode 94.5 tok/s, time to first token 0.77 s; CPU (4 threads) prefill 118 tok/s, decode 17.2 tok/s, time to first token 8.75 s (the mean of two passes each). The same numbers on the Galaxy S26's GPU with the Python API, one engine, no cache, from thermal status 1: prefill 275.87 tok/s, decode 20.09 tok/s, time to first token 3.77 s (the mean of runs 2 and 3 of 3).

The fp16 vision file

Mage-VL-Cowork_vision-fp16.litertlm is the main file with only its vision encoder replaced, for a vision encoder on the CPU: the model metadata reduced_precision_support = fp16accfp16 makes XNNPACK compute it in fp16, which took the vision stage on the Galaxy S26 from 39.86 s to 18.24 s (warm, from a cool phone in one session, these two encoders in builds with a shorter KV cache; on this file 18.83 and 19.08 s).

fp16 needed two changes, and neither moves the encoder's fp32 output (bit-equal on 11 pictures): the LayerNorms of layers 13–23 take their input times 2^-k and their eps times 4^-k (k up to 4), because their squared differences pass fp16's largest value (65,504); and the first LayerNorm (ln_pre) takes its input times 8 and its eps times 64, because on a flat screen its variance is an fp16 subnormal, the fp16 ADD with eps returned 0, RSQRT gave inf and the attention spread the NaN to every token. The fully connected weights and biases are stored as fp16 (a section of 613 MB against 1.22 GB), because LiteRT-LM 0.19.0's weight cache aborts the engine on the fp16 weights XNNPACK makes from fp32 ones unless the cache is off.

It is a separate file because only the CPU path uses it: with fp16 activations on the Mac GPU the fp32 encoder's output correlates at 0.599 with its fp32 output on the CPU, so the GPU devices keep the fp32 encoder, and we did not run this file's encoder on a GPU. In fp16 on the CPU the vision output is close to fp32's, not equal (correlation 0.977–0.985 on three real screenshots, 0.53–0.55 on single-colour pictures), so this file has its own test rows.

Where it does not run

• The vision encoder on the Galaxy S26's GPU. The main file's vision encoder, run alone with a LiteRT 2.2.0 CompiledModel, compiles with all 7,724 nodes on the GPU, and about 8 s later the Adreno driver ends the process (cl_a8x_cmdbuf_mgr_submit_ibs, exit code 137) with 3,785,360 kB (3.88 GB) of memory still available; attention chunks of 1,020 rows instead of 255 end the same way, about 19 s into the run. We did not run the whole file in this form.
• The vision encoder on the Galaxy S26's NPU. We compiled the encoder with the LayerNorm pre-scales for SM8850 (QAIRT 2.47, optimization level O2), with attention chunks of 255 and of 1,020 rows. The compiled graph loads, and its first execution crashes the compute DSP (SSR) on both graphs; so does an earlier build's bundle with the 1,020-row graph and the decoder on the CPU: 3 runs of 3. We did not try splitting the encoder into two graphs, 16-bit activations with 8-bit weights (SRQ a16w8) or optimization level O3. Neither file has an NPU form.
• Pixel 8a. Neither file was run on it. Two earlier builds with a shorter KV cache were, and both stopped while the engine was being created: our test stops a run at the first MemAvailable reading under 400,000 kB. The earlier build of the fp16 vision file read 374,600 kB (0.38 GB) 37 s in, with 3.54 GB of GPU memory and rising, before any vision stage; the other build read 359,820 kB (0.37 GB) with 3.99 GB. On the Pixel 11 Pro the 4,096-token KV cache took 0.88 GB more GPU process memory than the shorter one.
• The Pixel 11 Pro's Edge TPU. We did not try it.

Conversion notes

• Source: microsoft/Mage-VL-Cowork at revision 6fbe7f91 (4,741,793,792 parameters in bf16: 4,411,424,256 in the text model including the untied output head, 330,369,536 in the vision encoder and merger).
• Text decoder: the checkpoint's Qwen3-4B text model (36 layers, hidden 2560, 32 query and 8 key-value heads, vocabulary 151,936) as a standalone Qwen3ForCausalLM, exported with litert-torch 0.9.3, int4 weights (blockwise, 128 per block, OCTAV; a floor of 2^-14 on the block scales changed none) and an int8 embedding table outside the decoder for the fast_vlm path. Both files keep one prefill signature of 1,024 tokens and the decode signature with a 4,096-token KV cache, and the GPU's cache-update and attention composites were added to the graph. The decoder declares no activation type, so the GPU runs it in fp16. fp16 holds integers exactly only up to 2,048, and a request's positions go past it (to 3,451 on the 120 test screenshots, to 3,900 with 40 earlier-step lines), so the graph's rotary angles are computed from position = 2048·h + l with precomputed tables for the 2048·h part (the method and tool of LiteRT-LM #3913). On 12 test screenshots the fp16 decoder put the click in the target's box on 1 without that change (0 with the GPU composites added) and on 8 with it; the same decoder declared fp32 (before the composites) did on 7.
• Vision: the 24-layer encoder and the 2 × 2 merger for one static picture of 1920 × 1088 (68 × 120 patches, 2,040 image tokens), the patches in raster order with their rotary positions fixed, attention computed in chunks of 255 query rows so every tensor has at most 4 dimensions; fp32 weights and fp32 activations declared (with fp16 activations on the Mac GPU the encoder's output correlates at 0.599 with the CPU's fp32 output). The fp16 vision file's encoder is described above.
• Template: the source repository's chat template, rewritten for LiteRT-LM's template engine; the reply starts with <think> unless enable_thinking is false (not tested). Stop token <|im_end|>, no start token, no thought channel declared. The tokenizer is the repository's own tokenizer.json byte for byte.
• Reproduction: REPRODUCE.md · machine-readable facts: litertlm_manifest.json.

Known runtime issues

• Later turns. From the second turn of a conversation on, LiteRT-LM puts the previous answer's stop token into the KV cache with a zero embedding (these files hold the embedding table in their own section and the decoder takes embeddings). So later answers in a conversation can differ from the answer the model gives to the same messages in one prompt. The code that writes a token without an embedding (FillInputBufferWithToken, llm_litert_compiled_model_executor.cc) is the same in 0.16.0, 0.18.1 and 0.19.0 (read in the source); measured with 0.18.1 on WeVisOne-VL-2B and Qwen2.5-Coder-1.5B-Instruct, not on these files. A new conversation per inference, with earlier steps in its prompt, avoids it; that is what the script and the two steps above do.

License and changes

Apache-2.0, inherited from microsoft/Mage-VL-Cowork. Changes from the original work: exported to the LiteRT-LM format for one 1920 × 1088 picture; the text decoder quantized to int4 and its embedding table to int8; the decoder's rotary-position computation rewritten for fp16; the vision encoder's attention computed in chunks of 255 query rows; in Mage-VL-Cowork_vision-fp16.litertlm, the vision encoder's LayerNorm inputs scaled by powers of two, its fully connected weights and biases stored as fp16 and its metadata set for fp16 inference; the chat template rewritten for LiteRT-LM; stop token <|im_end|>. The sample picture samples/settings.png was drawn for this card with magevlcowork_work/make_sample.py of hf-to-litertlm (DejaVu fonts). Problems with these files: open a discussion in this repo.