Into the LidarVerse:
A Real-Time Unified LiDAR World Model Across Tasks and Sensors

Henry Che^1,2   Xuanchi Ren¹   Sanja Fidler¹   Shenlong Wang²   Laura Leal-Taixé¹   Shengyu Huang¹

¹NVIDIA    ²University of Illinois Urbana-Champaign

       

Quickstart | Fine-tuning | Model card

Abstract

LidarVerse is a real-time unified LiDAR world model for generation, point-cloud forecasting, and generalizable resimulation. It combines the flexibility of generative modeling with reconstruction-grounded simulation without per-scene optimization. An attention-based tokenizer maps different LiDAR scanning patterns into a shared latent space, while flexible conditioning on observed LiDAR, HDMaps, and accumulated geometry enables one model to perform all three tasks. Self-forcing distillation produces a four-step causal model; a rolling temporal K/V cache and a persistent spatial cache support long-horizon generation at 11 FPS on a single H100 GPU. LidarVerse achieves state-of-the-art performance on most reported metrics across the three evaluation protocols. The framework also supports cross-sensor generation with intensity, and its tokenizer generalizes to an unseen scanning pattern.

Stage 3 Quickstart

lidarverse-sfdmd-pai/ contains the EMA-only SF-DMD student. Use the current LidarVerse code and run from its repository root:

python scripts/download_checkpoints.py
CUDA_VISIBLE_DEVICES=0 STAGE=3 MODE=all NUM_SAMPLES=1 \
NUM_STEPS=4 SAMPLER=sde SDE_ETA=1 GUIDANCE=1 SEED=0 \
bash scripts/diffusion/getting_started.sh

The single-sample quickstart uses the second official NVIDIA PAI clip. Sampling uses four-frame blocks, KV9 including the sink, FP32 tokenizer encoding/decoding, and a persistent spatial cache with 0.10 m voxels. HDMap generation rasterizes raw lanes and actor edges with calibrated rays and rolling-shutter poses, then widens only bbox returns to 3x3 pixels before encoding. Set RAW_HDMAP_CONDITION_RASTERIZE=0 for the original cached condition; forecasting/resimulation do not use this HDMap override. Set PREPARE_DATA=0 to reuse prepared examples.

The package contains no regular-model, fake-score, optimizer, or training state. Fine-tuning from it starts a new run, not an exact full-state continuation. Its teacher came from the same causal-training lineage as the released Stage 2, but not from the exact Stage-2 snapshot packaged here.

LidarVerse Overview

Current checkpoint scope: the quickstarts use the range-only NVIDIA PAI tokenizer and PAI bidirectional, causal, and SF-DMD EMA packages, with 128x1024 inputs at 10 Hz and 2.5--100 m support. They do not include intensity or the paper's cross-sensor checkpoints. The broader architecture and dataset descriptions below describe the research project, not a claim that these particular packages support every sensor. Quickstart throughput is not a paper benchmark.

Description:

LidarVerse is a real-time unified LiDAR world model for sensor simulation across tasks and sensors. LidarVerse bridges both generative and reconstruction-based simulation into a single, unified conditional generative framework. It is composed of a universal, fully attention-based tokenizer maps LiDAR range sequences with different scanning patterns into a shared latent space; a bidirectional diffusion teacher learns a common interface for conditioning on a LiDAR prefix, an HD-Map, a persistent spatial cache, and optionally, text; and a four-step autoregressive causal student and maintain both a rolling temporal K/V cache and a world-aligned spatial cache, enabling persistent long-horizon roll-outs with real-time inference at 11 FPS on a single H100 GPU.
LidarVerse was developed by NVIDIA.
This model is for research and development only.

License/Terms of Use:

NVIDIA Open Model License

Deployment Geography:

Global

Use Case:

This model is intended for research community to study, ablate, and build upon of.

Release Date:

GitHub: 09/11/2026 via https://github.com/nv-tlabs/lidarverse

Reference(s):

Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Models

Model Architecture:

Architecture Type: Transformer
Network Architecture: Diffusion Transformer (DiT)
This model was developed based on Cosmos-Predict2-T2V-2B.
Number of model parameters: 2B

Input:

Input Type(s): Video
Input Format(s): Lidar
Input Parameters: Three-Dimensional (3D)
Other Properties Related to Input: Flexible resolution and length

Output:

Output Type(s): Video
Output Format: Lidar
Output Parameters: Three-Dimensional (3D)
Other Properties Related to Output: Flexible resolution and length, streaming supported

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.

Software Integration:

Runtime Engine(s): Not Applicable (N/A)
Supported Hardware Microarchitecture Compatibility:

• NVIDIA Ampere
• NVIDIA Blackwell
• NVIDIA Hopper
• NVIDIA Lovelace

Supported Operating System(s): Linux

The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.

Model Version(s):

LidarVerse v1.0 — 2B-parameter trained research model

Training, Testing, and Evaluation Datasets:

Training Dataset:

Data Modality:

• Video
• Text
• Image

Image Training Data Size: 1 Million to 1 Billion Images
Text Training Data Size: Less than a Billion Tokens
Video Training Data Size: 10,000 to 1 Million Hours
Data Collection Method by dataset: Automatic/Sensors
Labeling Method by dataset: Hybrid: Automatic/Sensors/Manually-Labeled Properties: 112,803 sensor-generated LiDAR range-sequence entries: NVIDIA PAI (111,988), PandaSet (93), and nuScenes (722). Sensor grids: NVIDIA PAI 128×1200 at 10 Hz, PandaSet 64×1024 at 10 Hz, and nuScenes 32×1024 at 20 Hz. No linguistic content.

Testing Dataset:

Data Collection Method by dataset: Automatic/Sensors
Labeling Method by dataset: Hybrid: Automatic/Sensors/Manually-Labeled Properties: 1,319 sensor-generated LiDAR range-sequence entries: NVIDIA PAI (1,180), PandaSet (10), nuScenes (128), and KITTI (1). Sensor grids: NVIDIA PAI 128×1200 at 10 Hz, PandaSet 64×1024 at 10 Hz, nuScenes 32×1024 at 20 Hz, and KITTI 64×1024 at 10 Hz. No linguistic content.

Evaluation Dataset:

Benchmark Score: Refer to Technical Report

Data Collection Method by dataset: Automatic/Sensors
Labeling Method by dataset: Hybrid: Automatic/Sensors/Manually-Labeled Properties: 202 sensor-generated LiDAR range-sequence entries from the Waymo Open Dataset validation split. Sensor grid: 64×1024 at 10 Hz. No linguistic content.

Inference:

Acceleration Engine: PyTorch Hardware Requirements (GPU Architecture, Model):

• "NVIDIA H100"
• "NVIDIA A100 80GB"

Ethical Considerations:

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. Developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
Please make sure you have proper rights and permissions for all input image and video content; if image or video includes people, personal health information, or intellectual property, the image or video generated will not blur or maintain proportions of image subjects included.
For more detailed information on ethical considerations for this model, please see the Model Card++ Bias, Explainability, Safety & Security, and Privacy Subcards. Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns here.

Generated by NVIDIA Model Card Generator Toolkit.

Citation

If you find LidarVerse useful for your research, please cite:

@misc{che2026lidarverse,
  title = {Into the {LidarVerse}: A Real-Time Unified {LiDAR} World Model Across Tasks and Sensors},
  author = {Henry Che and Xuanchi Ren and Sanja Fidler and Shenlong Wang and Laura Leal-Taix{\'e} and Shengyu Huang},
  year = {2026},
  eprint = {2610.19201},
  archivePrefix = {arXiv},
  primaryClass = {cs.CV}
}