SEV-UI-4B (E13)

SEV-UI-4B is a 4B typed-decision model for voice and chat control of a screen. It does not generate text. Given the current screen (as accessibility lines) and what the user has said so far, it returns a probability for each option of a multiple-choice question in one forward pass:

• readiness: can the computer act on what has been said so far? (act_now / wait / not_ui)
• pick: which listed screen element to click or type into
• boundary: after one request is done, where a second request starts in the remaining words
• type text, system command, open app

It is the fast path of a desktop agent: simple screen actions are decided by this model in about 0.1 s per question, and anything that needs planning or writing goes to an LLM agent. On a set of 19 site tasks, sending a command to seeing the screen change took about 1 s on this path versus a median of 10.6 s on the LLM agent path.

This checkpoint is experiment E13 (seed 7), the version currently served. It is Qwen3.5-4B with a LoRA adapter merged into the weights.

• Developed by: Plateer AI Lab (author: Seongjun Son)
• Base model: Qwen/Qwen3.5-4B
• Training and serving code: Eikos at commit f9dff4038238fbfee0afeaaa51f6f3208f705884

How it works

The model follows the Eikos typed-decision protocol. Each option gets a letter (A, B, C, ...). The prompt is rendered from the state, the question and the options, and the probabilities are the softmax (T = 1) of the letter logits at the first answer token. The answer cannot fall outside the options.

Because options are letters A–Z, one question can hold at most 26 options. Screens with more elements are asked in groups of 25, and the group winners are asked again in a final.

Usage

The model must be called through the Eikos server, which renders the prompt exactly as in training. Plain chat prompting will not give these results.

vLLM 0.30.0 or newer is required. Qwen3.5 is a hybrid (Gated DeltaNet) architecture. With vLLM 0.27.1 we measured 90.25% on our English benchmark with one request at a time but 86.05% with 16 concurrent requests. vLLM 0.30 gave 90.3% with 16 concurrent requests.

git clone https://github.com/caiovicentino/eikos && cd eikos
git checkout f9dff4038238fbfee0afeaaa51f6f3208f705884

# engine: letter readout + hybrid prefix cache
bash scripts/serve_vllm.sh /path/to/SEV-UI-4B 8001

# decision API in front of the engine (reads decision_config.json from the model folder)
python eikos/serve.py --model /path/to/SEV-UI-4B --vllm-url http://127.0.0.1:8001 --port 8000

State format

Put the screen first and the user's words last. Only the last line changes as speech comes in, so vLLM's prefix cache computes the screen once per screen.

Page: https://shop.example
Screen:
e1 | link | "로그인" | 오른쪽 위
e2 | textbox | "검색어를 입력하세요" | 가운데 위
e3 | link | "이벤트" | 왼쪽 가운데
User said: 검색창에 운동화

• One line per element: id | role | "name" | position. Use at most 60 lines and cut names at 32 characters.
• The position is the 3 × 3 cell of the element's center: 왼쪽/가운데/오른쪽 and 위/가운데/아래. The exact center is 가운데. Positions were trained only where frames were known, so you can leave them out.
• If the target may be beyond 60 lines, ask over windows of 60 lines. Do not move a likely target into the first 60 lines.

Questions

Several questions can share one request and one state.

{
  "state": "Page: https://shop.example\nScreen:\ne1 | link | \"로그인\" | 오른쪽 위\ne2 | textbox | \"검색어를 입력하세요\" | 가운데 위\ne3 | link | \"이벤트\" | 왼쪽 가운데\nUser said: 이벤트 보여줘",
  "questions": {
    "ready": {
      "type": "choice",
      "instructions": "Can the computer act on what the user has said so far?",
      "criteria": {
        "act_now": "yes — what to do is clear now",
        "wait": "not yet — the user has not finished saying what to do",
        "not_ui": "no screen action — a question, chat, or something to write"
      }
    },
    "pick": {
      "type": "choice",
      "instructions": "Which listed screen element should be acted on (clicked or typed into) to carry out what the user said?",
      "criteria": { "e1": "e1", "e2": "e2", "e3": "e3" }
    }
  }
}

Send it with POST /v1/systemone. Each answer has choice, confidence and probabilities.

The question texts used in training are:

question	instructions	options
readiness	Can the computer act on what the user has said so far?	the three options above
pick	Which listed screen element should be acted on (clicked or typed into) to carry out what the user said?	element ids, at most 25 per group (labels are the ids; names stay in the screen lines)
boundary	In "<utterance>", the first request ("<words of the first request>" …) is done. Where does a second, different request start?	none: there is no second request — the remaining words only finish the first one, and w<i>: starts at "<word>" for each remaining word
type text	Should the word "<w>" (or its stem) appear in the text that gets written? and Is the word "<w>" part of the text the user wants typed, rather than a command word?	yes / no, averaged over the two forms; the typed span grows from the top word while p(yes) ≥ 0.8
system	Which system command carries out the user's instruction?	system commands
open app	Which application does the user want to open?	app names, at most 25 per group

Correcting position bias. Option order can still shift the probabilities. In the final round, ask the same candidates in forward and reverse order in one request and average the two distributions.

Training

family	rows	source
pick	20,237	AndroidControl train episodes (8,000, positions from screen bounds), Korean teacher commands (7,237), web element instructions (5,000, no positions)
type field / type text	1,755 / 2,549	AndroidControl, Korean teacher commands
system	3,975	AndroidControl train (2,000) and the same kind of command rewritten as spoken Korean (1,975)
open app	500	AndroidControl
readiness	6,300	prefixes before and after the first actionable word (wait 3,000, act_now 2,000), non-screen requests (not_ui 1,300)
boundary	2,500	two commands on one screen joined with a connector such as 그리고 or then (1,500); commands ending in politeness words, gold none (1,000)
Eikos public sample	697	caiovicentino1/eikos-decisions, mixed in to keep the general decision skill

Total: 38,513 rows.

• AndroidControl: train episodes only. Validation and test episodes are used only for evaluation.
• Korean teacher commands: a teacher model (DeepSeek) wrote spoken-style Korean commands for elements on accessibility snapshots of 115 public Korean websites. A second model chose the element from the command alone, and we kept only the commands it got right (96%). The readiness marks, the not_ui requests and the spoken Korean system commands were also written by DeepSeek.
• Web element instructions: Qwen3.8-27B wrote instructions for elements on public websites, then chose the element back from the instruction. We kept the round-trip matches.
• Exclusions: no evaluation item, site or screen is in the training data.

Recipe (Eikos defaults, view augmentation and JEPA losses off):

• LoRA r = 64, alpha = 128, dropout 0.05, all linear layers
• AdamW, lr 1e-4, 1 epoch, batch 2 × 4 gradient accumulation, max length 3,072 tokens
• loss: soft cross-entropy on the option-letter logits, with option order shuffled in every row
• auxiliary rationale LM loss × 0.3 (up to 192 tokens, not used at inference)
• one RTX 5090, about 2 h 25 min per run (4,333 steps)

Evaluation

Every comparison was preregistered: models, data splits, metrics, pass criteria and stop conditions were committed before any result was inspected. The baseline is the previous served model, E10. Unless noted, E10 is the mean of seeds 7, 11 and 13, and E13 is the mean of seeds 7 and 11: the seed 13 run was killed from outside the process twice, and the preregistration allowed one retry.

benchmark	E10	E13
Korean speech, real time (67 commands recorded by one speaker after preregistration; transcript fed word by word)	75.6%	88.8%
acted on a wrong target mid-speech, same set	11.9%	0.8%
same set through a production streaming STT (seed 7)	77.6%	88.1%
two commands in one utterance, 52 items (seed 7, STT)	51.9%	71.2%
DecisionBench v1: AndroidControl test/validation episodes, 2,000 items (pick, type, open app, system)	0.891	0.907
web53: 53 hand-written web instructions	0.912	0.953

Latency (seed 7, vLLM):

• last word to action, median: 558 ms (E10: 345 ms). E13 waits for the Korean main verb, and E10 was partly fast because it acted too early.
• a single question on a small screen: about 0.1 s
• end of audio to action through a streaming STT: about 0.9 s for both models; the STT's re-transcription cycle dominates

Limitations

• Unseen sites. On 17 public sites and 106 commands not used anywhere in development, the rate was 65.1%, well below the benchmark numbers above. Of the 37 failures, 14 came from the executor (pop-ups covering elements, input) and 23 from the model.
• Absent targets. When the requested element is not on the screen, the model picks the most similar element instead of abstaining. On 9 such commands it abstained 0 times and picked with confidence 0.94–1.00. A follow-up run trained to abstain learned it but started missing present targets, and was not adopted.
• Over-confidence. Wrong answers can carry 0.98–0.99. Do not treat confidence alone as a safety gate.
• Ordinal references. References such as 첫 번째 링크 ("the first link") are still often wrong.
• Narrow human evaluation. The Korean speech sets were recorded by one speaker.
• Text only. Inputs are accessibility lines, not screenshots. The vision weights of the base model are included unchanged.
• Out of scope. The model writes nothing and cannot plan. Confirm irreversible actions, such as deleting, paying or submitting, outside the model.

Data and code availability

Only the merged weights are released. The training rows, the evaluation sets, and the code that builds the data and runs training are not released. The tables above describe what they contain. Serving uses the public Eikos code at the commit given above.

License and attribution

• Model weights: Apache-2.0, following the base model Qwen/Qwen3.5-4B.
• Training and serving code: Eikos (MIT, Copyright 2026 Caio Vicentino).
• Training data: AndroidControl (Apache-2.0); a 697-row sample of eikos-decisions (CC BY 4.0); teacher data generated with DeepSeek (whose platform terms permit training on outputs) and Qwen3.8-27B (Apache-2.0).

Citation

@misc{sev-ui-4b-2026,
  title  = {SEV-UI-4B: a typed-decision model for real-time voice control of screens},
  author = {Son, Seongjun},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/xgen-ai/SEV-UI-4B}}
}