DriveIK: Grounding Driving VLA via Inverse Kinematics
NeurIPS 2026
Junsung Park, Hyunjung Shim β KAIST, Kim Jaechul Graduate School of AI
Official NAVSIM checkpoints of DriveIK, a camera-only driving vision-language-action (VLA) model built on Qwen2.5-0.5B-Instruct.
Driving VLA models often plan blindly: on ego-predictable examples the planner can rely on ego status and the driving command instead of the scene. DriveIK anchors trajectory decoding on visual states. The LLM predicts the future visual state of the scene, and a separate Inverse Kinematics (IK) network decodes the trajectory from only the current and predicted visual states, so it cannot fall back on ego status or text.
Checkpoints
| Folder | Stage | Initialized from | Epochs | NAVSIM-v1 PDMS β | NAVSIM-v2 EPDMS β |
|---|---|---|---|---|---|
DriveIK_NAVSIM_sft |
World model + IK (SFT) | Qwen2.5-0.5B-Instruct | 30 | 90.3 | 89.3 |
DriveIK_NAVSIM_rft |
GRPO with inline PDM rewards (RFT) | DriveIK_NAVSIM_sft |
10 | 92.2 | 90.6 |
Both models are trained on NAVSIM navtrain and evaluated on navtest. Scores are from the paper; see it for the full tables and ablations.
Model
- LLM: Qwen2.5-0.5B-Instruct (24 layers, hidden size 896) in a LLaVA-style layout.
- Visual input: the three front cameras (
CAM_L0,CAM_F0,CAM_R0). Each is resized to 288Γ160 and encoded by a frozen DINOv3 ViT-S/16, which gives 180 patch tokens per camera and 540 Γ 384 tokens per frame. A linear projector maps them to the LLM width. No LiDAR or BEV encoder is used. - Prompt: ego status, the 2 s trajectory history, the mission goal and the previous action, as text.
- Next visual state prediction: the LLM regresses the 540 visual tokens of the frame 4 s ahead.
- IK network: a cross-attention conditional diffusion head with 50 sampling steps. It decodes 8 waypoints (4 s at 0.5 s spacing) from the current and predicted visual states.
Files
Each folder contains:
| File | Contents |
|---|---|
model.safetensors |
All trained weights (~2 GB, fp32): LLM, visual and ego projectors, IK head |
config.json, generation_config.json |
Model configuration (model_type: llava_qwen) |
ik_mode.json |
IK head configuration |
vocab.json, merges.txt, tokenizer_config.json, special_tokens_map.json, added_tokens.json |
Tokenizer with the DriveIK special tokens |
The DINOv3 encoder is not included. It is a gated model: accept its license on the
model page and log in with huggingface-cli login. The code downloads it on first use.
These checkpoints use a custom model class and IK head. Load them with the DriveIK code,
not with transformers.AutoModel.
Usage
Get the code from github.com/engineerJPark/DriveIK
and download the checkpoints into its checkpoints/ folder:
git clone https://github.com/engineerJPark/DriveIK.git && cd DriveIK
huggingface-cli download engineerJPark/DriveIK --local-dir checkpoints
# or a single stage:
huggingface-cli download engineerJPark/DriveIK --include 'DriveIK_NAVSIM_rft/*' --local-dir checkpoints
Evaluate a checkpoint with the launcher of the same stage. The two recipes differ in ego-feature width and precision, and a mismatched launcher gives much worse trajectories.
# SFT
DATA_ROOT=/data/driveik \
SENSOR_ROOT=/data/navsim/sensor_blobs/trainval VAL_SENSOR_ROOT=/data/navsim/sensor_blobs/test \
EVAL_CKPT=checkpoints/DriveIK_NAVSIM_sft \
bash train_navsim/DriveIK_NAVSIM_sft.sh
# RFT
DATA_ROOT=/data/driveik \
SENSOR_ROOT=/data/navsim/sensor_blobs/trainval VAL_SENSOR_ROOT=/data/navsim/sensor_blobs/test \
EVAL_CKPT=checkpoints/DriveIK_NAVSIM_rft \
bash train_navsim/DriveIK_NAVSIM_rft.sh
To run the RFT stage from the released SFT model, pass
SFT_CKPT=checkpoints/DriveIK_NAVSIM_sft to train_navsim/DriveIK_NAVSIM_rft.sh. See the
code repository for installation, data
layout and training.
Inputs and limitations
- The models are trained only on NAVSIM (OpenScene / nuPlan) driving logs with three front cameras. Other camera setups, sensor layouts or regions are untested.
- These are research checkpoints for benchmark evaluation. They are not meant to control a real vehicle.
License
The DriveIK weights are released under the Apache License 2.0. The base model Qwen2.5-0.5B-Instruct is also Apache 2.0. DINOv3 is distributed separately under its own license, and the NAVSIM / nuPlan data are subject to their own terms.
Citation
@inproceedings{park2026driveik,
title = {Grounding Driving VLA via Inverse Kinematics},
author = {Park, Junsung and Shim, Hyunjung},
booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
year = {2026}
}