RoboJev-27B

RoboJev-27B estimates the probability that a robot completed a task. The input is the task instruction and four frames from one camera, taken at the start of the attempt, one third and two thirds of the way through, and at the end. The output is P(completed). A request takes about 0.5 s on one H100.

It is JEV-27B-VL fine-tuned on robot success labels.

Examples

Two SO-101 attempts at "Insert the colourful ring into the central wooden peg", a task held out from training. RoboJev was queried every 0.5 s with the frames recorded up to that point (robojev_overlay.py).

Intended uses

• Scoring policy evaluation rollouts.
• Success reward for reinforcement learning on real robots.
• Filtering failed or incomplete demonstrations from datasets.
• Checking each step when a program controls a robot.

Evaluation

All models received the same frames and the same question.

ArmnetBench, SO-101 (armnet/armnetbench_v01_robometer): 268 rollouts from seven policies across eight tasks, labelled by people. RoboJev's training data includes other recording sessions from this setup; two of the eight tasks are held out entirely.

Model	AUROC	Balanced accuracy	ECE	AUROC, held-out tasks
RoboJev-27B	0.987	0.924	0.052	0.973
JEV-27B-VL	0.983	0.877	0.113	0.977
TypeSafe Jev 1.14	0.975	0.829	0.177	0.958
Perplexity Decider v1.2	0.954	0.811	0.116	0.947
Inception Mercury Decide	0.938	0.816	0.109	0.949
Liquid d1-ultra	0.918	0.814	0.062	0.878
Robometer-4B (whole video)	0.777	0.677	0.251	0.812

Unseen labs (robometer/rbm-1m-ood): 459 rollouts from eight labs (Franka, Koch, Trossen, xArm and SO-101 arms). None of these labs is in the training data.

Model	AUROC	Balanced accuracy	ECE
Perplexity Decider v1.2	0.885	0.761	0.168
JEV-27B-VL	0.880	0.774	0.162
RoboJev-27B	0.872	0.761	0.173
Liquid d1-ultra	0.859	0.735	0.143
Inception Mercury Decide	0.859	0.756	0.205
TypeSafe Jev 1.14	0.841	0.721	0.197
Robometer-4B (whole video)	0.818	0.598	0.414

AUROC is the probability that a random success scores higher than a random failure. Balanced accuracy is the mean of the recall on successes and on failures at a 0.5 threshold. ECE is the expected calibration error over 10 bins.

On the unseen labs the top three models are within noise: with 459 episodes, AUROC differences below about 0.03 are not significant. In 28 of these episodes all four top models confidently contradict the label. In the eight we inspected, the recording ends before the outcome is visible. The list is in benchmark/bench_ood_v1_disputed.jsonl.

Usage

vllm serve josefchen/RoboJev-27B --served-model-name robojev --max-model-len 32768 \
  --limit-mm-per-prompt '{"image": 8}' --logprobs-mode processed_logprobs --trust-request-chat-template

from robojev import RoboJev

judge = RoboJev("http://localhost:8000")
judge.done("Put the ring on the peg", "attempt.mp4")                          # samples four frames
judge.done("Put the ring on the peg", ["a.jpg", "b.jpg", "c.jpg", "d.jpg"])  # or pass them

done returns P(completed). robojev.py needs requests, pillow and, for video files, ffmpeg. The model needs one 80 GB GPU in bf16.

To draw the probability on a video or a camera feed:

python robojev_overlay.py "Put the ring on the peg" attempt.mp4 out.mp4
python robojev_overlay.py "Put the ring on the peg" --camera 0       # requires opencv-python

lerobot_plugin/ adds a robojev reward model to LeRobot, for recorded episodes, and a reward step for real-robot training that replaces the per-task reward classifier (see lerobot_plugin/README.md):

hf download josefchen/RoboJev-27B --include "lerobot_plugin/*" --local-dir robojev
pip install ./robojev/lerobot_plugin

The model does not generate text. The prompt ends in [decision]:; the client reads the log-probabilities of the tokens false and true at that position, adds the biases in decision.json, divides by the temperature and applies a softmax.

Files

• Weights (bf16, 54.7 GB): Qwen3.8-27B with JEV-27B-VL's decision LoRA and the RoboJev LoRA merged in. Loads with vllm serve or transformers like any Qwen3.8 checkpoint.
• decision.json: prompt format, token ids, biases and temperature.
• robojev.py: client. robojev_overlay.py: draws the probability on a video or camera feed.
• lerobot_plugin/: LeRobot reward model and reward step.
• examples/: the two videos above.
• benchmark/: both test sets, the disputed episodes and the scores of every model.

Training

The base model is JEV-27B-VL (Apache-2.0), which is Qwen3.8-27B with a LoRA for typed decisions. That LoRA was merged into the weights before training.

The training data combines the ArmnetBench training sessions with 19,000 episodes from Robometer's RBM-1M collection. The RBM-1M part has real rollouts from RoboArena, AutoEval, SOAR, RoboReward and FINO-Net (about half of them failures), simulated rollouts from LIBERO and Meta-World, and demonstrations from 17 Open X-Embodiment and MolmoAct datasets. None of it comes from the labs in the unseen-lab evaluation. Negative examples include episodes truncated at one third to one half of their length and successful episodes paired with another task's instruction. Frames were randomly rescaled, recompressed and colour-jittered.

We trained one epoch over 33,000 examples with log loss and a rank-16 LoRA on the language model, keeping the vision encoder frozen. This took 3.6 hours on one H200. The released weights apply the LoRA update at scale 0.5 (WiSE-FT); we fixed that scale before evaluating on the unseen labs. The temperature was fitted on held-out episodes from the training sources.

Limitations

• Weakest on wrist-mounted cameras and long-horizon tasks such as stirring or folding.
• It only sees the frames it is given. If a recording ends before the outcome, the answer is unreliable.
• With only the last frame as input it is weaker on new robots (AUROC 0.779 on the unseen labs).
• It is not a safety system.

License

The weights are Apache-2.0, as are JEV-27B-VL and Qwen3.8. Training data licences: ArmnetBench (Apache-2.0), RoboArena (MIT), RoboReward (CC BY 4.0), MolmoAct (CC BY 4.0), LIBERO and Meta-World (MIT), Open X-Embodiment subsets (mostly CC BY 4.0). SOAR, AutoEval and FINO-Net have no stated data licence; Robometer's processed copies were used.