NIDAR

NIR-Guided Intrinsic Decomposition for Scalable Scene-Agnostic
LiDAR Intensity Reconstruction

IROS 2026 · Official model artifacts

Paper · Code · Project page · Demo guide

NIDAR adds synthesized intensity to an existing point cloud using RGB appearance, a pseudo-NIR prior, intrinsic decomposition, and calibration. It requires RGB images + XYZ geometry + camera/LiDAR calibration.

Ground-truth intensity	NIDAR intensity

Existing nuScenes research example from the project page. This figure is not a validation run of the new official-data converter.

Download And Use

File	Role	Local runtime destination
47.pth	RGB-to-pseudo-NIR STN	stn/47.pth
epoch_010.pth	U-Net + DeepLab IRNet	irnet/epoch_010.pth
wintensity_mapping_quantile_model.pkl	Waymo quantile calibration	quantile/wintensity_mapping_quantile_model.pkl
deep_intrinsic_public_model.yaml	Matching IRNet architecture	Select through the inference runner

Runtime destinations are relative to the chosen weights directory; files on the Hub are at the repository root. ARTIFACTS.json and SHA256SUMS provide sizes and checksums.

Release preview: the matching GitHub source is updated. A two-frame CPU acceptance run completed in 27.4 seconds on a ThinkPad, with full anonymous weight downloads, GT-independence checks, and an Open3D window test. Obtain Waymo and nuScenes data from their official providers, then use the conversion guide. Waymo's official two-frame raw TFRecord now also passes conversion and complete CPU inference (3.83 s conversion; 29.79 s demo). Two official nuScenes mini scene-0061 keyframes also passed conversion, full CPU inference (28.31 s), SDK projection checks and saved-image inspection. See the Waymo raw-fixture report and nuScenes two-frame report. These small checks do not establish full-segment/split quality or CUDA support. The Open3D window test above used the earlier Waymo input, not nuScenes.

Follow the source demo guide using the current GitHub source for the new converters. The earlier acceptance used source revision 8803e880e08662b07409f4bc9a9f6fb487101c99. Python 3.10 and PyTorch 2.6 or newer are required by the reviewed inference package; checkpoint release checks use PyTorch 2.8. This is a custom PyTorch pipeline. Training reproduction is a follow-up release; UE5 and Isaac Sim demos are deferred.

Prepare Data And Run

We do not host copies of Waymo or nuScenes. Use the provider's own account and download process, then convert locally:

• Waymo guide: download the small two-frame TFRecord directly from Waymo's official SDK repository, or use your own Perception segment. TOP LiDAR first return.
• nuScenes guide: official mini split, two keyframes with six cameras and LIDAR_TOP.

After preparing the input and the source inference environment:

python tools/download_weights.py --weights-root ./weights
python inference/run_nidar_demo.py \
  --input-root ./data/waymo_demo --weights-root ./weights \
  --output-root ./outputs/waymo_demo --viewer auto

For nuScenes, change the input to ./data/nuscenes_demo and use a new output directory. The export manifest selects the correct calibration. The nuScenes remap file is bundled in the GitHub source, not hosted in this model repository. Use --check-input before inference, --device cpu for CPU execution, or --viewer none on a headless machine. No training is needed for the demo.

Expected outputs include scalar-intensity NPY/PLY, a colorized PLY, a static preview, and GT/prediction comparisons when evaluation targets are present. The optional Open3D viewer opens an interactive point-cloud window locally.

Model Route

RGB -> STN pseudo-NIR -> IRNet R -> LiDAR-camera projection
    -> quantile remapping -> intensity point cloud

The default output uses R + remap, rather than the direct intensity head I_hat. The selected STN has local training evidence based on the cs-stereo architecture and is frozen for inference. Its metadata records selected epoch 47 of 48 configured epochs, batch size 16, learning rate 5e-5, and PittsStereo-RGBNIR sequence IDs. See PROVENANCE.md for the audit evidence and remaining limits.

Checkpoint downloads can pin revision 1390055c2c564d3276f29cc513b3ef3cdc355e40. Companion files were added later and must be fetched from a commit that contains them. Check sizes and SHA-256 before loading trusted checkpoints or calibration pickles. Matching hashes establish identity, not a security certification.

Scope And Limitations

NIDAR generates an intensity-like research signal, not absolute returned optical power, sensor-independent material reflectance, or new XYZ geometry. Exposure, spectral appearance, range, visibility, projection, and sensor shift can affect outputs. This model repository hosts the Waymo calibration; the source also includes a separate nuScenes calibration. Neither is sensor-independent. Intrinsic outputs can blur boundaries or vary between adjacent frames.

The paper contains research evaluations. Later controlled ablations use separately trained models and are not fresh benchmarks of these hosted files. No new manipulation-success claim is established by this artifact upload. The CPU checks validate the inference workflow on small two-frame inputs; the real nuScenes check covers one scene only. They do not establish generalization, GPU performance, or robotic task success.

License And Attribution

The code, weights, calibration, and data have distinct terms. Read LICENSE_NOTICES.md and the retained upstream cs-stereo MIT notice. Final training-data and weight distribution reconciliation is pending; this card grants no blanket permissive or commercial weight license. No raw training dataset is included.

Citation And Contact

@misc{zhang2026nidar,
  author = {Junjie Zhang and Jie Yin and Kefei Qian and Jie Li and
            Mengpei Jia and Yajuan Dun and Wenbo Chu and Guofa Li},
  title = {{NIDAR}: {NIR}-Guided Intrinsic Decomposition for Scalable Scene-Agnostic {LiDAR} Intensity Reconstruction},
  year = {2026},
  eprint = {2609.36878},
  archivePrefix = {arXiv},
  primaryClass = {cs.RO},
  url = {https://arxiv.org/abs/2609.36878}
}

Questions: GitHub Issues. Project contact: Junjie Zhang.