ZeroVLN
ZeroVLN: Learning to Navigate without Supervised Fine-Tuning of the VLM Brain
ZeroVLN is a vision-and-language navigation model that follows natural-language instructions in continuous environments without navigation-domain supervised fine-tuning of its vision-language model (VLM) brain. At each interaction round, the brain describes the intended movement, a continuous-action expert executes it, and the brain uses the resulting observation to decide whether to move again or stop.
The action expert first learns from reference trajectories while the base brain remains frozen. Closed-loop, multi-turn reinforcement learning then updates both policies from trajectory returns. After joint training, the brain largely retains its general visual capabilities. Its mean accuracy across 12 visual benchmarks with free-form answers is 59.16%, compared with 59.47% for the base brain.
Checkpoint contents
| Path | Role |
|---|---|
qwen/ |
Jointly trained Qwen3.6-35B-A3B brain, tokenizer and processor. The brain weights are stored in 16 BF16 safetensors shards. |
learnable_query/learnable_query_embeddings.pt |
The 16 learnable query embeddings. Their contextual representations condition the action expert on the brain’s movement request. |
action_head/ |
Continuous-action expert weights and architecture configuration. |
report.json |
Export metadata used to validate the model files. |
training_config.json |
Training settings associated with this checkpoint, used by the evaluation launcher. |
The full download is about 70.4 GB. The expert also uses a separate frozen Qwen3.5-4B visual backbone. Supply its local path and the benchmark scenes and episode files when configuring the runtime.
Load the checkpoint
Start from a checkout of the ZeroVLN code repository. Complete the environment setup, install the Hugging Face CLI, and set FAST_BACKBONE and the benchmark asset paths in scripts/env/environment.local.sh. From the code repository root, run:
source ./scripts/env/environment.local.sh
export PATH="$(dirname "$PYTHON"):$PATH"
export INITIAL_CHECKPOINT="$RL_ARTIFACT_ROOT/checkpoints/ZeroVLN"
# If access is restricted, authenticate with an account that can read this repository.
hf download hzxllll/ZeroVLN --local-dir "$INITIAL_CHECKPOINT"
"$PYTHON" scripts/prepare_checkpoint.py "$INITIAL_CHECKPOINT"
export SLOW_MODEL="$INITIAL_CHECKPOINT/qwen"
export TRAINING_CONFIG="$INITIAL_CHECKPOINT/training_config.json"
./scripts/eval/run_navigation.sh --config configs/gpu/evaluation.json \
--output-dir "$RL_ARTIFACT_ROOT/navigation-console" --port 5002
The preparation command makes a local compatibility link for the learnable query weights and can be run again. Open the console on port 5002, select a scene and instruction, then run an episode. The code Quick start pins the exact Hub revision.
Evaluation
The table reports success rate (SR) and success weighted by path length (SPL), both in percent, for ZeroVLN in closed-loop simulation.
| Benchmark | SR ↑ | SPL ↑ |
|---|---|---|
| SAGE-3D | 51.3 | 32.0 |
| R2R-CE | 52.7 | 29.2 |
| RxR-CE | 51.4 | 28.1 |
These results cover simulated navigation on the three listed benchmarks. Loading the brain alone provides only the high-level policy. Reproducing the navigation results also requires the learnable queries, action expert, frozen visual backbone and benchmark runtime.