OmniJev 2B · 全模态 Jev
Beijing Zhongguancun Academy · Institute of Automation, Chinese Academy of Sciences · Zevo
A decision layer on top of Qwen/Qwen3.5-2B that answers typed questions (choice / noul / score) about images, screens,
game frames, video, sound (as spectrogram images) and robot views with calibrated probabilities in one forward pass,
0 generated tokens. Code and usage: https://github.com/tinnel123666888/OmniJev (from mso.infer import MSO1;
MSO1("ckpt", "base") with this folder as ckpt and the backbone as base; the hybrid backbone is served through
the branch path automatically).
Files
adapter_config.json / adapter_model.safetensors, head.pt (decision head), ord.pt (ordinal head),
head_meta.json (head settings + per-type temperatures noul 1.00 / choice 1.02 / score 1.32), new_tok_emb.pt,
processor and tokenizer files. ~179 MB.
Training data
Web operation (Mind2Web), grid pointing (web + phone), AndroidControl phone operation (10k steps, 96-cell tap grid), Simple-Wikipedia navigation, LIBERO-10 robot decisions, real-robot manipulation (UT-Austin MUTEX, 1,500 episodes), LIBERO-Spatial, Charades-STA video events, long video / planning / spatial families, a real-time catch game, Atari human play (11 games), Snake, Gomoku, chess from real games with local candidate moves, HaGRID gestures + fire/smoke/weapons, ESC-50 sounds rendered as spectrograms, a self-built mixed decision set (regions, existence, phone, chess, short video). About 270,000 decision records and 1.3 million typed questions, balanced across domains and identical for all three released sizes; packed multi-question training under proper scoring rules; temperatures fitted on held-out rows.
Held-out accuracy (serving path)
| family | Qwen3.5-2B zero-shot | OmniJev 2B | OmniJev 4B (v0.5, Qwen3-VL) |
|---|---|---|---|
| LIBERO-10 robot decisions | 0.299 | 0.724 | 0.821 |
| Mind2Web test (task / website / domain) | 0.257 | 0.631 | 0.770 |
| Grid pointing, 96 cells (web) | 0.370 | 0.625 | 0.729 |
| Charades-STA video events | 0.481 | 0.826 | 0.864 |
| Catch game frames | 0.170 | 0.728 | 0.885 |
| HaGRID gestures + fire/smoke/weapons | 0.584 | 0.984 | 0.974 |
| OK-VQA answer pool | 0.831 | 0.765 | 0.893 |
| LongVideoBench val | 0.568 | 0.502 | 0.588 |
| Long video / planning / spatial | 0.385 | 0.529 | 0.686 |
| v0.1 image families | 0.584 | 0.623 | 0.819 |
| Wiki navigation | 0.405 | 0.667 | 0.701 |
| Real-robot MUTEX | 0.261 | 0.687 | – |
| Atari human play | – | 0.691 | – |
| Snake | – | 0.763 | – |
| Gomoku | – | 0.669 | – |
| Chess (local candidate moves) | – | 0.609 | – |
| AndroidControl phone operation | – | 0.668 | – |
| ESC-50 sounds (spectrogram) | – | 0.514 | – |
Latency on one idle A800-40G (the serving path, a 768-token image budget, median of 12 runs with bench/speed_bench.py): one question in 217 ms, twelve questions about the same image in 277 ms, i.e. 23.1 ms per question. All three released sizes were measured back to back on that one card, so the numbers can be compared with each other.
License: Apache-2.0 for the weights and the code; the backbone keeps its own license.