StreetNVS: Effective Multi-sensor Conditioning for Street-view Novel-view Synthesis

Zhengfei Kuang¹ · Adam Sun¹ · Liyuan Zhu¹ · Tong Wu¹ · Shengqu Cai¹ · Jonathan Tremblay² · Iro Armeni¹ · Ehsan Adeli¹ · Lior Yariv¹ · Gordon Wetzstein¹

¹ Stanford University    ² NVIDIA

This is the official model release for StreetNVS: Effective Multi-sensor Conditioning for Street-view Novel-view Synthesis.

Modern vehicle platforms are equipped with a rich sensor suite, including LiDAR, calibrated multi-camera rigs, and accurate ego-motion, that in principle offers strong signal for re-rendering a driving scene from novel viewpoints. A growing line of recent work leverages video diffusion models for this task, using their generative priors to synthesize plausible novel views from sparse vehicle observations. In practice, however, existing methods exploit only a fragment of this signal, and their quality tends to degrade as the target trajectory departs from the recorded driving path. We argue that this is fundamentally a multi-sensor fusion problem: sparse LiDAR reprojections supply accurate but incomplete metric geometry, surround-view reference imagery supplies dense appearance but no metric depth, and camera poses tie the two together across views. We introduce StreetNVS, a video diffusion framework that jointly conditions on all three signals through a Reference-Enhanced Camera Attention module based on a relative ray-level positional encoding. We develop a two-stage curriculum training strategy that gradually exposes the model to increasingly sparse LiDAR. On the Waymo Open Dataset, StreetNVS substantially outperforms state-of-the-art baselines under sparse LiDAR conditioning, matches methods that rely on 10–100× denser point clouds. We further show capabilities of synthesizing coherent videos along extreme out-of-trajectory paths such as elevation, lane-shift, pullback, and rotation.

Novel-view Synthesis

StreetNVS combines sparse LiDAR, surround-view images, and camera poses to generate new views of a driving scene. Here are two examples from our paper, using camera paths that move well beyond the recorded views.

Bird's-eye view	Extreme lane shift

Watch bird's-eye view · Watch extreme lane shift

The videos above are hosted on our project page, where you can also compare LiDAR conditioning, different sampling ratios, and baseline methods.

Model

streetnvs.safetensors contains the complete StreetNVS diffusion model. It is used with the text encoder, VAE, and tokenizer from Wan2.2-TI2V-5B.

Code and running instructions are in StreetNVS/StreetNVS. Follow the data preparation guide to prepare your own inputs through the Waymo Open Dataset.

Citation

@article{kuang2026streetnvs,
  title={Effective Multi-sensor Conditioning for Street-view Novel-view Synthesis},
  author={Kuang, Zhengfei and Sun, Adam and Zhu, Liyuan and Wu, Tong and Cai, Shengqu and Tremblay, Jonathan and Armeni, Iro and Adeli, Ehsan and Yariv, Lior and Wetzstein, Gordon},
  journal={arXiv preprint arXiv:2606.01590},
  year={2026}
}

Acknowledgements

We build on Wan2.2, DiffSynth-Studio, and PRoPE. Thank you to the authors for sharing their code and models. We also thank Astra for helping prepare this release.

StreetNVS inference extensions are released under the Apache License 2.0. Third-party preprocessing and rasterizer components retain their own license terms, including research/non-commercial restrictions; see NOTICE. Model and dataset licenses remain those of their respective providers.