OmniAgent-RL-7B

OmniAgent-RL-7B is the final checkpoint of OmniAgent, the first native omni-modal agent for active perception in video understanding (ICML 2026). It is built on Qwen2.5-Omni-7B and trained in two stages — Agentic SFT, then Agentic RL with TAURA. This is the recommended checkpoint and reproduces the main results in the paper.

• 📄 Paper: Native Active Perception as Reasoning for Omni-Modal Understanding
• 💻 Code: https://github.com/HarryHsing/OmniAgent
• 🤗 Models: OmniAgent-RL-7B · OmniAgent-SFT-7B

What it does

Instead of consuming every frame, OmniAgent runs an Observation–Thought–Action (OTA) loop: a single omni model decides what to look at (get_frames), listen to (get_audio), or watch (get_clip) on demand, distills each percept into a compact textual memory, and answers when it has enough evidence. The environment only returns raw media — all perception and reasoning are done by this model.

How to use

⚠️ This is an agent checkpoint. The weights follow the Qwen2.5-Omni-7B architecture, but to reproduce OmniAgent's active perception you must run it inside the OTA environment in the GitHub repo, not via a plain transformers call.

# After setup (see the repo README):
bash demo/launch_inference.sh checkpoints/OmniAgent-RL-7B assets/example_video_mcq.mp4

Model size & components

OmniAgent uses only the Qwen2.5-Omni thinker for agent reasoning; the talker (audio generation) is never used, and only the thinker is loaded and used during training and inference. The text-to-audio tag is auto-derived from the Qwen2.5-Omni architecture in config.json and does not reflect how OmniAgent is used.

Which checkpoint should I use?

• OmniAgent-RL-7B (this model) — best performance; use for inference, evaluation, and deployment.
• OmniAgent-SFT-7B — the cold-start checkpoint; use it to re-run Agentic RL yourself or to study the SFT-only stage.

Results (highlights)

• Open-source SoTA across 10 video / audio-visual / temporal-grounding benchmarks.
• On LVBench, this 7B agent outperforms the 10× larger Qwen2.5-VL-72B (50.5 vs. 47.3) with ~73% fewer frames.
• Large temporal-grounding gains over the Qwen2.5-Omni-7B base (+33 IoU on LongVALE and VUE-TR).

See the paper for the full tables.

Limitations

The sequential OTA loop adds inference latency compared to a single forward pass; reducing this via parallel exploration is left to future work.

Citation

@inproceedings{xing2026omniagent,
  title={Native Active Perception as Reasoning for Omni-Modal Understanding},
  author={Zhenghao Xing and Ruiyang Xu and Yuxuan Wang and Jinzheng He and Ziyang Ma and Qize Yang and Yunfei Chu and Jin Xu and Junyang Lin and Chi-Wing Fu and Pheng-Ann Heng},
  booktitle={International Conference on Machine Learning (ICML)},
  year={2026}
}