ullaltushar/qwen-drive-thor

Qwen-Drive-1.0-4B — Jetson AGX Thor Deployment

This repository contains the Qwen-Drive-1.0-4B planning pipeline deployed on Jetson AGX Thor (p3960 board, sm_110, CUDA 13.3, TensorRT 11.0.1.7, Edge-LLM 0.11.0).

Model Overview

Qwen-Drive-1.0-4B is a Vision-Language-Action (VLA) model for autonomous driving planning. It takes driving scenes (camera images + ego history + navigation command) and outputs a 50-waypoint trajectory (x, y, heading in metres/radians) using a two-stage process:

Stage	Component	Engine	Precision	Size
Stage 1	VLM Prefill + KV Cache Dump	llm.engine	NVFP4 (W4A8)	~3.8 GB
Stage 1	Visual Encoder	visual.engine	fp16	~671 MB
Stage 2	Planning Expert	action.engine	fp16	~2.0 GB

Architecture:

• VLM: Qwen3.5-4B with visual encoder (Qwen3.5-VL)
• Expert: Flow-matching diffusion transformer, 10-step Euler integration
• Trajectory: 50 waypoints at 10 Hz, 5.0 second horizon
• Coordinate frame: Ego vehicle at t=0 (x forward, y left, heading positive = left turn)

Hardware Constraints

Jetson AGX Thor has 6.4 GB GPU-addressable memory (measured). The VLM engine (3.8 GB) and action expert (2.0 GB) cannot coexist, so the pipeline runs as two sequential processes:

1. Stage 1: llm_inference — VLM prefill → KV cache dump to disk
2. Stage 2: qd_sample — Flow-matching expert → trajectory

Repository Contents

File	Description
llm.engine	VLM TensorRT engine (NVFP4, 3.8 GB)
visual.engine	Visual encoder TensorRT engine (fp16, 671 MB)
action.engine	Planning expert TensorRT engine (fp16, 2.0 GB)
llm_config.json	VLM engine config
visual_config.json	Visual engine config
multimodal_config.json	Multimodal runner config
tokenizer.json	Qwen tokenizer
tokenizer_config.json	Tokenizer config
embedding.safetensors	FP8 embedding table
chat_template.jinja	ChatML chat template

Usage

Prerequisites on Thor

# Environment variables
export LD_LIBRARY_PATH=/usr/local/cuda-13.3/lib64:/usr/lib/aarch64-linux-gnu:${LD_LIBRARY_PATH:-}
export EDGELLM_PLUGIN_PATH=/mnt/vblkdev2/edgellm-011/build/libNvInfer_edgellm_plugin.so

# Binaries
#   /mnt/vblkdev2/edgellm-011/build/examples/llm/llm_inference
#   /mnt/vblkdev2/qd_sample

Run Inference

# Stage 1: VLM prefill → KV dump
export QD_SIDECAR_JSON=/path/to/sidecar.json
export QD_KV_DIR=/path/to/kv/output

llm_inference \
  --engineDir /path/to/llm/ \
  --multimodalEngineDir /path/to/multimodal/ \
  --inputFile /path/to/request.json \
  --outputFile /tmp/out.json \
  --batchSize 1 --maxGenerateLength 1

# Stage 2: Expert → trajectory
qd_sample \
  --engine /path/to/action.engine \
  --kvDir /path/to/kv/output \
  --sidecar /path/to/sidecar.json \
  --out /path/to/trajectory.json \
  --steps 10

Output Format

{
  "trajectories": [
    [
      [0.10, -0.06, -0.016],
      [0.41, -0.14, -0.059],
      ...
      [17.56, -12.26, -0.417]
    ]
  ]
}

50 waypoints, each [x(m), y(m), heading(rad)] in ego frame at 10 Hz.

Validation Results

Tested on 3 demo scenes from Qwen-Drive dataset:

Scene	Intent	Speed	Mean 2D Error	Max Heading Error	Status
Scene 0	GO_STRAIGHT	~0 m/s	1.22 m	29.2°	✅ PASS
Scene 2	GO_RIGHT	2.7 m/s	2.54 m	2.5°	✅ PASS
Scene 3	GO_STRAIGHT	7 m/s	4.29 m	0.3°	✅ PASS

Cross-platform validation (Thor vs H200 bf16 reference):

• KV cosine similarity: 0.974
• Trajectory max 2D error: 1.27 m, mean 0.47 m (scene 0)

License

Apache-2.0

Contact

Thor host: visteon@10.138.192.161