Create README.md

README.md

@@ -0,0 +1,198 @@
+---
+license: apache-2.0
+language:
+- en
+tags:
+- autonomous-driving
+- vision-language-action
+- chain-of-thought
+- trajectory-prediction
+- VLA
+base_model:
+- Qwen/Qwen3-VL-4B-Instruct
+pipeline_tag: image-text-to-text
+---
+
+# OneVL: One-Step Latent Reasoning and Planning with Vision-Language Explanation
+
+**[📄 Paper (arXiv)](https://arxiv.org/abs/2604.18486)** | **[💻 GitHub](https://github.com/xiaomi-research/onevl)** | **[🌐 Project Page](https://Xiaomi-Embodied-Intelligence.github.io/OneVL/)**
+
+*Xiaomi Embodied Intelligence Team*
+
+---
+
+## Overview
+
+**OneVL** is a Vision-Language-Action (VLA) framework for autonomous driving that achieves **state-of-the-art trajectory prediction accuracy** while matching the inference latency of answer-only autoregressive models.
+
+Prior latent Chain-of-Thought (CoT) methods compress reasoning into opaque hidden states — fast, but consistently underperform explicit CoT on driving tasks. OneVL identifies the root cause: purely linguistic latents encode abstract semantic labels rather than the spatiotemporal causal dynamics that govern real driving scenes. OneVL addresses this with **dual-modal auxiliary decoders** that force compact latent tokens to encode both human-readable reasoning *and* future scene dynamics simultaneously.
+
+At inference, both decoders are discarded and all latents are **prefilled** into the prompt context in a single parallel pass — matching answer-only AR prediction speed while recovering the interpretability of explicit CoT in both vision and language.
+
+OneVL is the **first latent CoT method to surpass explicit autoregressive CoT** across all four driving benchmarks.
+
+---
+
+## Architecture
+
+OneVL augments **Qwen3-VL-4B-Instruct** with three components:
+
+**Latent Token Interface** — 4 visual latent tokens + 2 language latent tokens are inserted in the assistant response before the answer, using existing vocabulary tokens (no new special tokens added).
+
+**Visual Auxiliary Decoder** — Predicts future-frame visual tokens at t+0.5s and t+1.0s from visual latent hidden states (using the Emu3.5 IBQ 131k codebook). Acts as a **world model** supervision signal that forces the latent space to encode genuine physical scene dynamics — agent trajectories, road geometry, and environmental change — rather than abstract descriptions.
+
+**Language Auxiliary Decoder** — Reconstructs explicit CoT reasoning text from language latent hidden states, conditioned on ViT visual features. Recovers 97% of explicit CoT text quality while running at answer-only speed.
+
+**Prefill Inference** — Both decoders are discarded at inference time. All latent tokens are processed in a single parallel prefill pass; only the trajectory answer is generated autoregressively. This achieves **1.5× speedup over explicit CoT on NAVSIM** and **2.3× on ROADWork**.
+
+### Three-Stage Training Pipeline
+
+Training proceeds in three stages to ensure stable joint optimization:
+- **Stage 0**: Main model warmup (trajectory prediction)
+- **Stage 1**: Auxiliary decoder warmup (language + visual decoders independently)
+- **Stage 2**: Joint end-to-end fine-tuning (all components together)
+
+Staged training is essential — ablation shows that skipping it collapses PDM-score from 88.84 to 67.13.
+
+---
+
+## Results
+
+### NAVSIM
+
+| Method | Model Size | PDM-score ↑ | Latency (s) ↓ | Interpretability |
+|---|:---:|:---:|:---:|:---:|
+| AR Answer | 4B | 87.47 | 4.49 | — |
+| AR CoT+Answer | 4B | 88.29 | 6.58 | Language |
+| COCONUT | 4B | 84.84 | 5.93 | — |
+| CODI | 4B | 83.92 | 8.62 | — |
+| SIM-CoT | 4B | 84.21 | 10.86 | Language |
+| **OneVL** | **4B** | **88.84** | **4.46** | **Vision + Language** |
+
+### ROADWork
+
+| Method | ADE (px) ↓ | FDE (px) ↓ | Latency (s) ↓ |
+|---|:---:|:---:|:---:|
+| AR CoT+Answer | 13.18 | 29.98 | 10.74 |
+| **OneVL** | **12.49** | **28.80** | **4.71** |
+
+### Impromptu
+
+| Method | ADE (m) ↓ | FDE (m) ↓ | Latency (s) ↓ |
+|---|:---:|:---:|:---:|
+| AR CoT+Answer | 1.42 | 3.96 | 6.84 |
+| **OneVL** | **1.34** | **3.70** | **4.02** |
+
+### APR1 (Alpamayo-R1)
+
+| Method | ADE (m) ↓ | FDE (m) ↓ | Latency (s) ↓ |
+|---|:---:|:---:|:---:|
+| AR CoT+Answer | 2.99 | 8.54 | 3.51 |
+| **OneVL** | **2.62** | 7.53 | **3.26** |
+
+### CoT Text Quality (NAVSIM)
+
+| Method | Meta Action Acc. ↑ | STS Score ↑ | LLM Judge ↑ | Latency (s) ↓ |
+|---|:---:|:---:|:---:|:---:|
+| AR CoT+Answer | 73.20 | 79.75 | 81.86 | 6.58 |
+| **OneVL** | 71.00 | 78.26 | 79.13 | **4.46** |
+
+OneVL's language auxiliary decoder recovers 97% of explicit CoT quality at answer-only inference speed.
+
+---
+
+## Usage
+
+### Requirements
+
+- Python 3.10+, CUDA GPU (≥16 GB VRAM recommended)
+- `transformers >= 4.57.0` (required for `Qwen3VLForConditionalGeneration`)
+
+```bash
+uv venv venv/onevl --python 3.12
+source venv/onevl/bin/activate
+pip install -r requirements.txt
+```
+
+### Inference (Trajectory Prediction Only)
+
+```bash
+python infer_onevl.py \
+    --model_path /path/to/OneVL-checkpoint \
+    --test_set_path test_data/navsim_test.json \
+    --image_base_path "" \
+    --output_path output/navsim/results.json \
+    --device cuda:0 \
+    --num_latent 2 --num_latent_vis 4 \
+    --max_new_tokens 1024 --answer_prefix "[" --prefix_k 0
+```
+
+### Inference with Language + Visual Explanation
+
+```bash
+python infer_onevl.py \
+    --model_path /path/to/OneVL-checkpoint \
+    --test_set_path test_data/navsim_test.json \
+    --image_base_path "" \
+    --output_path output/navsim/results_explain.json \
+    --device cuda:0 \
+    --num_latent 2 --num_latent_vis 4 \
+    --max_new_tokens 1024 --answer_prefix "[" --prefix_k 0 \
+    --decoder_explain --aux_visual_condition \
+    --c_thought 2 --max_explain_tokens 1024 \
+    --visual_decoder_explain --visual_aux_visual_condition \
+    --c_thought_visual 4 --max_visual_tokens 2560
+```
+
+### Multi-GPU Inference
+
+```bash
+export MODEL_PATH=/path/to/OneVL-checkpoint
+export TEST_SET_PATH=test_data/navsim_test.json
+export OUTPUT_PATH=output/navsim/navsim_results.json
+bash run_infer.sh
+```
+
+Per-benchmark scripts are available in `scripts/`:
+
+```bash
+bash scripts/infer_navsim.sh
+bash scripts/infer_ar1.sh
+bash scripts/infer_roadwork.sh
+bash scripts/infer_impromptu.sh
+```
+
+For full documentation, evaluation scripts, and data format details, see the [GitHub repository](https://github.com/xiaomi-research/onevl).
+
+---
+
+## Open-Source Status
+
+| Component | Status |
+|---|:---:|
+| Technical Report | ✅ Released |
+| Model Weights | ✅ Released |
+| Inference Code | ✅ Released |
+| Training Code | 🔜 Coming Soon |
+
+---
+
+## Citation
+
+```bibtex
+@article{lu2026onevl,
+  title={OneVL: One-Step Latent Reasoning and Planning with Vision-Language Explanation},
+  author={Lu, Jinghui and Guan, Jiayi and Huang, Zhijian and Li, Jinlong and Li, Guang and Kong, Lingdong and Li, Yingyan and Wang, Han and Xu, Shaoqing and Luo, Yuechen and others},
+  journal={arXiv preprint arXiv:2604.18486},
+  year={2026},
+  url={https://arxiv.org/abs/2604.18486}
+}
+```
+
+---
+
+## License
+
+Released under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0).
+
+Model weights are built on [Qwen3-VL-4B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct) and the visual tokenizer is from [Emu3.5-VisionTokenizer](https://huggingface.co/BAAI/Emu3.5-VisionTokenizer); please refer to their respective licenses as well.