laya-cua

One repo, two Laya checkpoints. Root = computer-use action type (AgentNet). function_calling/ = call or no-call and which function (Glaive). Both are decision layers: they do not write text, coordinates or arguments, and they do not read screenshots.

A small (421M) model for computer-use agents. Given a task, a text description of the current screen and the last few steps, it answers two questions in one forward pass:

  1. What kind of action comes next? One of: click, type, key, scroll, move, wait, drag, done.
  2. Is the task already finished? (yes/no probability)

What it does NOT do

  • It does not generate text, so it cannot write the string to type.
  • It does not output click coordinates or element IDs.
  • It does not read screenshots. The input is text (see format below).
  • It is not a full agent. Use it as a fast router or sanity check next to whatever picks the target and the text (an LLM, a grounding model, a script).

Input and output

Input: a dict with task (instruction), screen (text description or accessibility tree of the current screen, trained on up to 1,800 characters) and history (up to the last 4 steps as short strings). Output: answers["action_type"]["choice"] is one action name, answers["task_done"]["noul"] is the probability the task is done. The result also carries confidence values.

Usage

pip install laya huggingface_hub
import laya
from huggingface_hub import snapshot_download


ACTIONS = {
  "click": "click or tap a UI element", "type": "type text into a focused field",
  "key": "press a key or shortcut", "scroll": "scroll the view",
  "navigate": "open a URL or switch app or tab", "move": "move the mouse without clicking (hover)",
  "wait": "wait for the page or app to load", "drag": "drag and drop or select by dragging",
  "done": "the task is already complete, stop",
}
QUESTIONS = {
  "action_type": {"type": "choice", "instructions": "Which action type should the agent take next?", "criteria": ACTIONS},
  "task_done":   {"type": "noul",   "instructions": "Is the task already fully complete on screen?"},
}
agent = laya.Agent(snapshot_download("sodan/laya-cua"), device="cuda")  # or "cpu"
state = {"task": "Open the settings app and turn on dark mode",
         "screen": "Settings window. Sidebar: Network, Display, Sound. Display page shows Theme: Light.",
         "history": ["opened Settings"]}
res = agent.predict(state, QUESTIONS)
print(res["answers"]["action_type"]["choice"], res["answers"]["task_done"]["noul"])

The calls (laya.Agent, predict, answers[...]) are the same ones used for the evaluation below, and loading this repo from the Hub was run and worked. The exact snippet above is being re-run end to end and this line will be updated once it has.

Training

  • Base: convaiinnovations/laya (ModernBERT-large encoder plus a typed decision head). Method: the official laya.train RLCD loop on one free Colab T4.
  • Data: OpenCUA AgentNet (xlangai/AgentNet, MIT), 400 tasks, one row per step: 5,822 train rows. Held-out: the same 334 rows (from 200 other tasks, whole tasks held out) used for every version. Labels come from the recorded action. needs_human has no label and was not trained.
  • 2 epochs, same recipe as v0. The only change between v0 and v1 is more training tasks.

Held-out results (n=334, single run each)

Held-out action_type accuracy, n=334, single run each. Decision layer only: no coordinates, text or vision.

metric v1 (current) v0 (200 tasks) base Laya always "click"
action_type accuracy 0.8084 0.7665 0.1856 0.7156
task_done accuracy 0.9611 0.9581 0.7066 not computed

Results log

version change action_type task_done status
v0 200 tasks, 2 epochs 0.7665 0.9581 replaced
v1 400 tasks, 2 epochs 0.8084 0.9611 current (re-downloaded from this repo and re-evaluated: same numbers)

Function calling checkpoint (function_calling/)

convaiinnovations/laya fine-tuned on glaiveai/glaive-function-calling-v2 to decide (1) whether to call a function now and (2) which listed function. It does not write function arguments. Load it the same way as the main model, pointing laya.Agent at the function_calling folder after snapshot_download("sodan/laya-cua") (this exact path was not re-run after the move from a separate repo; the files are byte-identical copies).

Function-calling results

Split by conversation: train 4500 rows, dev 1454, test 2574. 2 epochs, free Colab T4, one run, one seed.

Test metric Base Laya Fine-tuned Always no-call Random pick
tool_call accuracy (n=2574) 0.3042 0.9573 0.648 n/a
tool_choice accuracy (n=906) 0.9989 1.0 n/a 0.8786

Dev: tool_call 0.2978 (base) vs 0.9519 (fine-tuned). The gain is the call/no-call decision; tool_choice is near the ceiling for base Laya because most prompts list one or two functions. Single run, no confidence intervals. An earlier run on laya 0.4.37 gave test tool_call 0.9545.

Live check on unseen AgentNet steps (Colab T4, one run)

Streamed AgentNet tasks far past the 400 used for training and eval (469 steps from 30 tasks). action_type accuracy 0.859 vs 0.744 for always-click. Small sample, steps are clustered by task, and the task mix differs from the n=334 eval above, so do not compare the two numbers as a trend. In 8 randomly shown examples 6 matched; the misses were VS Code steps (predicted click, recorded move or key). Rare actions (move, key) are mostly missed.

Negative result: Mind2Web zero-shot

osunlp/Mind2Web (CC-BY-4.0) train split, 80 tasks, 628 steps, never seen by Laya. Screen = tags stripped from cleaned_html, first 1500 characters (far from the prose descriptions Laya trained on). Task: CLICK / TYPE / SELECT.

Mind2Web zero-shot

model op accuracy
always CLICK 0.8376
laya-cua v1 0.8041 (TYPE recall 2/86)
base Laya 0.1513

v1 does not beat always-CLICK here and does not transfer to raw web text. Laya has no SELECT class, so 16 SELECT steps can never match. Single run.

Limits

  • Small test set (334 rows), a single run per version, only 400 tasks of training data.
  • v1 is about 9 points above always predicting "click" (0.8084 vs 0.7156). That is a measured gap on this one split, not a guarantee elsewhere.
  • About 94% of held-out rows are "not done", so task_done accuracy is close to a trivial baseline.
  • Trained on AgentNet text descriptions (mostly Ubuntu-style desktop tasks). Behavior on other screen formats, apps or phone UIs is untested.
  • Besides the Mind2Web zero-shot test above, no other benchmarks were run. Laya is a decision layer, not an end-to-end computer-use agent.

License: Apache-2.0 (as the base). Data: AgentNet, MIT.

Downloads last month
39
Safetensors
Model size
0.4B params
Tensor type
F16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for sodan/laya-cua

Finetuned
(258)
this model

Dataset used to train sodan/laya-cua

Space using sodan/laya-cua 1